fixed length file manipulation - is there a better way?

[Scott Williams_5]

we have many data files that are fixed-length files with fixed-length fields. I was tasked with masking out some or all of two fields within each record. I wrote a script that used "cut" to split the file into multiple slices (columns), then used sed to manipulate the data, and "paste" to paste the pieces back together.

I'm thinking that there's got to be a better way.

I've attached the script I used as my solution -- what would your solution have been?


Cheers

Scott

[A. Clay Stephenson · Accepted solution]

My cut at this would have been a Perl script; it would easily do all of this and can handle long records w/o LF's. It would execute at least 100X faster than your cut-based solution.

Your other option (as long as the records are 3000 characters or less) is awk. It too would be much more efficient. The Gnu version of awk, gawk, can handle arbitrarily large records to get around the 3K limit.

Learn Perl; that's my first choice but I ain't gonna code it for you because that's really doing you no favor.

[Scott Williams_5]

Thanks A. Clay -- I don't the script, I just need some good advise, and you have provided that nicely

Thanks

[H.Merijn Brand (procura]

use perl and pack/unpack
no line length limit

foreach my $filename (glob "flatord.*") {
open my $out ">$filename.new" or die "$filename.new: $!";
local @ARGV = ($filename);
while (<>) {
my ($f1, $nr, $f2, $dt, $f3) = unpack "a768 A16 a160 A4 a*", $_;
$nr = sprintf"%s%04d","X"x16,$nr%10000;
$dt =~ s/./X/g;
print $out "$f1$nr$f2$dt$f3";
}
close $out;
print "="x40,"\n"


Should be about the same, uses no temp files, is a single bloody fast process. Didn't test

Enjoy, Have FUN! H.Merijn

[H.Merijn Brand (procura]

Clay, I cannot type that fast .....

:)

And I also head to read and understand the attachment

Enjoy, Have FUN! H.Merijn