awk/sed script question

[Belinda Dermody]

I need to print a line to a error file and delete it at the same time using awk or sed in a shell script. The file is over 3million lines and I do not want to read it twice. The search string is response: 2

One additional situation is there a way that it could be multiple search strings ex: response: 2 and starting: 3...

This needs to be in a shell script, I am from the old school and havent found time to get proficient with Perl although I do a little. I can write them faster in shell.

[Leif Halvarsson_2]

Hi
I don't understand your first question. What is the meaning with printing a line to a file and delete it at the same time ?

You can use multiple search strings with grep

"grep -e "response: 2" -e "starting: 3" "

[Rodney Hills]

If you the following in scan.awk-

/response: 2/{print $0 >/tmp/otherfile;next}
/response: 3/{print $0 >/tmp/otherfile;next}
{print}

Then ran the following-

awk -f scan.awk yourinput >/tmp/remain.txt

Then /tmp/otherfile will have the "response" lines and /tmp/remain.txt would have the lines that are not "reponse" lines.

HTH

(by the way- perl is worth the investment to learn...)

-- Rod Hills

[Belinda Dermody]

I am sorry, with the size of the file over 3million lines, I would like to beable to print the line to another file (error.rpt) and then delete it from the original file. I would like to only make one pass of the original file. Because then I have to process the orginal file without the error lines for a final report.

[Ralph Grothe]

Hi James,

you can use sed | grep | awk | perl for this

e.g. supressing lines containing string "time"

sed '/time/d' /etc/services

You can use several expressions either by using multiple -e flags, or joining statements by ';'

This sould work for all parsers.

Afaik, you need to redirect the output in a new file unless you use Perl where you can use Perl's "in place edit" flag "-i"
please, type "perldoc perlrun"

[Gregory Fruth · Accepted solution]

If you really insist on using awk for this, perhaps
this will do what you want:

awk '{if ($0 ~ /^response: 2$/) {print > "errorlog"} else {print}}' input

Matching lines are printed to the file "errorlog" while
non-matching lines are printed to stdout, which you
can redirect as appropriate.

Note that this leaves the original file intact. (Don't try
to redirect the output of the awk command to the input
file.) If you're operating on a flat text file I don't see any
way of actually "deleting" a line in the file without doing
any copying.

HTH

[Shannon Petry]

You wont be able to do what you want without perl or coding some C, and even then it may not be easy.

1. Is this file open by any application when your trying to move lines to the error file? If so, good luck unless you can get the file to let go of this big file while your edit runs.
2. C and Perl would both be easy, as your going to open 2 files, and copy a string from file1 to file2, then replace the string in file1 with nothing.

awk, sed, and grep can not work with the file as is, as they are stream based editors. you will have to redirect output to a different location.
You would have to work with ed (I think) to in line edit the file, and even then if the file is open by another application you will have a hard time.

Hope the logic helps, and sorry I cant give a magic bullet.

Shannon

[Leslie Chaim]

I feel Shannon is right on this one, and the following Perl solution does act like a 'stream' but you can appreciate its brevity:)

perl -ni -e 'BEGIN{open ERRORS, ">error.rpt"} if ( /response: 2/ || /starting: 3/ ) {print ERRORS;next}print' your_BIG_text_file

This does an in-place edit of your_BIG_text_file, and creates an error.rpt file. If you have any doubts, then be safe and add extensions to the -i option above.

Cheers,
Leslie