topic Re: Finding and correcting curropt file lines in Operating System - HP-UX

Finding and correcting curropt file lines

Declan Heerey — Thu, 21 Aug 2003 15:37:27 GMT

I have a file where each line starts with one of the following strings;

HDR
POL
OBJ

I have a script which searches the file and errors if one of the lines has is corrupt. The scripts run as follows

cat | while read line
do grep -n -e "^HDR" -e "^POL" -e "^OBJ" -v > err_log.txt
done

I then manually edit the script at the lines reported in the err_log.txt (if any exist)

Is there any way I can get the script to perform the corrections for me? any help appreciated

Declan

Re: Finding and correcting curropt file lines

Massimo Bianchi — Thu, 21 Aug 2003 15:41:12 GMT

Hi,
i think that you have no luck, because corruption can have different method for appearing.

If you know exactly what kind of corruption can be, there are chances.

Other way, if you want to tear away the offending lines, you can use a nice grep like your, without "-v", redirecting output to a file, but i suspect some missing things.

Any more hints from you regarding the kind of corruption ?

Massimo

Re: Finding and correcting curropt file lines

Umapathy S — Thu, 21 Aug 2003 15:44:54 GMT

Declan,
You havnt said what to replace with. Corruption is random. If you want you can redirect all the valid lines to another file but you may be missing the corrupted lines which needs your manual attention.

-Umapathy

Re: Finding and correcting curropt file lines

Declan Heerey — Thu, 21 Aug 2003 15:47:02 GMT

Apologies i should have explained; corruption normally takes place when a line breaks before finishing and so begins another line i.e

HDR more text>
POL
OBJ

I then go to offending line (end of HDR line) and hit Ctrl J

Re: Finding and correcting curropt file lines

Erik Heckers — Thu, 21 Aug 2003 15:50:55 GMT

Hello!

I guess your 'grep' is already doing the trick if you remove -n and -v.

Or use this shorter solution:

grep -E "^(HDR|POL|OBJ)" repaired.txt

Erik

Re: Finding and correcting curropt file lines

TOMAS BERNABEU — Thu, 21 Aug 2003 15:51:03 GMT

Hi
use sed command for delete/change this lines.

To

Re: Finding and correcting curropt file lines

curt larson_1 — Thu, 21 Aug 2003 15:51:33 GMT

a start would be

cat $yourFile |
while read line
do
case $line in
HDR*) if corrupt ;then
fix
fi
print fixedLine;;
POL*) if corrupt ;then
fix
fi
print fixedLine;;
OBJ*) if corrupt ;then
fix
fi
print fixedLine;;
*) print $line;;
done > newFile

#check your fixes or
mv newFile yourFile

your have to put in how to test for corruption
and what the fix would be

Re: Finding and correcting curropt file lines

curt larson_1 — Thu, 21 Aug 2003 16:18:34 GMT

you could try something like this

cat yourFile|
awk '
#assume the first line starts correct
#print it without a newline
NR == 1 {printf("%s",$0);next;}
# if the line match our pattern
# apply the newline and
#print it without a newline
/^HDR/ {printf("\n%s,$0);next;}
/^POL/ {printf("\n%s,$0);next;}
/^OBJ/ {printf("\n%s,$0);next;}
# doesn't match one of out patterns
# print the line, joining it to the
# previous line
{printf("%s,$0);}
# apply an ending newline if necessary
END {printf("\n");}
'> newfile

you can apply spacing on the join and ending newline as desired

Re: Finding and correcting curropt file lines

Declan Heerey — Fri, 22 Aug 2003 07:44:54 GMT

Curt, that looks like the solution i need, however when i run the script against the file in question i get the following error messages;

awk: The string
%s,$0);ne cannot contain a newline character.
The source line is 13.
The error context is
/^HDR/ {printf("\n%s,$0);next;} >>>
<<<
syntax error The source line is 14.
awk: The statement cannot be correctly parsed.
The source line is 14.
awk: There are 3 missing } characters.
awk: There are 3 missing ) characters.
./test_and_fix[15]: Syntax error at line 15 : `(' is not expected.

Any ideas?

Re: Finding and correcting curropt file lines

Massimo Bianchi — Fri, 22 Aug 2003 09:24:20 GMT

Roland's answer might work, with proper " !! :)

cat yourFile|
awk '
#assume the first line starts correct
#print it without a newline
NR == 1 {printf("%s",$0);next;}
# if the line match our pattern
# apply the newline and
#print it without a newline
/^HDR/ {printf("\n%s",$0);next;}
/^POL/ {printf("\n%s",$0);next;}
/^OBJ/ {printf("\n%s",$0);next;}
# doesn't match one of out patterns
# print the line, joining it to the
# previous line
{printf("%s",$0);}
# apply an ending newline if necessary
END {printf("\n");}
'> newfile

One day or the other i will study awk seriusly....

Massimo

Re: Finding and correcting curropt file lines

Declan Heerey — Fri, 22 Aug 2003 09:38:40 GMT

One day i will do the same, until then any additional help would be appreciated. With the correct ""'s i get the following error;

awk: Cannot find or open file match.
The source line number is 17.
./test_and_fix[15]: Syntax error at line 15 : `(' is not expected.

Re: Finding and correcting curropt file lines

Charles G. — Fri, 22 Aug 2003 09:52:45 GMT

Hello Declan,

The following perl script will do what you expect :

#!/usr/bin/perl

# read the file and place it in a list
while (<>) {
chomp;
push(@liste,$_);

}

# print first line assumed ok
$line = shift(@liste);
print $line;

# handle other lines
foreach $line (@liste) {
# print a newline if the line did not match the patterns
print "\n" if ($line =~ /^HDR/ || $line =~ /^POL/ || $line =~ /^OBJ/);
print $line;
}

# print trailing newline
print "\n"

Save the script in a file xx.pl (eventually change the first line to match the path of a perl interpreter on your system), make the xx.pl file executable and use :

cat | xx.pl > file.out

Re: Finding and correcting curropt file lines

Declan Heerey — Fri, 22 Aug 2003 10:49:56 GMT

Charles, you're a star, that perl script works a treat!!! thanks for all the help guys, much appreciated.

I'm going to get onto my boss and sort out a perl course!

Great stuff and thanks again! I'll be able to relax over the weekend

Have a good one

Declan

Re: Finding and correcting curropt file lines

curt larson_1 — Fri, 22 Aug 2003 14:05:39 GMT

Declan

Massimo has pointed out my missing
double qoutes in the printf's

and

you might already satisfied with Charles's
perl script.

but it still looks like there was a syntax
issue with your script before that. If your
still interested in finding out what it was,
if you'd post your verison, i'm sure we can
figure out where the problem is.