Awk scripting Question

[Belinda Dermody]

How can I match a line in a log file "matched 1" and print that line plus the next 10 lines for a report. The input file will be daily and close to 2.4million lines. The information will always be in groups of 11

[Ian Dennison_1]

It'll take a while using awk.

cat /tmp/file1 |awk 'BEGIN {found=0;count=0}
found == 1 && count < 10 {print $0;count=count+1}
found == 1 && count >= 10 {found=0;count=0}
$1 ~ /matched1/ {found=1;count=1;print $0}
END'

Share and Enjoy! Ian

[curt larson_1 · Accepted solution]

awk '
/match 1/ {
print $0;
for (i=1;i<11;i++) {
getline;
print $0;
}
}

[Pete Randall]

Here's an example that could be adapted (taken from the attached "Handy One-Liners for Sed", courtesy of Princess Paula)

# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h


Pete

[harry d brown jr]

#!/usr/bin/ksh
#
awk 'BEGIN { lineno=0; goodlineno=0; }
{
# increment line number
lineno+=1;
# does it match our pattern?
if (match($0,/^matched 1/)) {
goodlineno=lineno;
print $0;
} else {
if ((goodlineno) && (lineno < (goodlineno+11))) {
print $0;
} else {
goodlineno = 0;
}
}
}
'


Though I think this is a JOB for perl.

live free or die
harry

[harry d brown jr]

I agree that perl will be a lot faster, and here is the awk script converted to perl:

#!/opt/perl/bin/perl
#
$lineno = 0;
$goodlineno = 0;
while (<>) {
# increment line number
$lineno += 1;
# does it match our pattern?
if ($_ =~ /^matched 1/ ) {
$goodlineno = $lineno;
print $_;
}
else {
if (($goodlineno) && ($lineno < ($goodlineno + 11))) {
print $_;
}
else {
$goodlineno = 0;
}
}
}


live free or die
harry

[Belinda Dermody]

Thanks for all the quick responses, Took the fewest lines first to use and it produced the correct output, I haven't checked the others. What they say, use as little as possible to get the best results.

Harry Brown, I am just starting to fool around with perl, where do you enter the file name to parse to get your output in the perl program.

[Robin Wakefield]

Hi James,

here's a perl one-liner:

perl -ne '{$n=11 if /matched 1/;$n--,print $_ if $n;$n=0 if eof}' logfile

Rgds, Robin.

[harry d brown jr]

James,

like I would in awk:

cat FILENAME | ./scriptname

Though I really like Robin's solution - quick and easy!

live free or die
harry

[H.Merijn Brand (procura]

Robin, that's good! Also catches overlaps. Can be shorter:

perl -ne '/matched 1/&&$n=11;--$n&&print' logfile

Enjoy, have FUN! H.Merijn

[harry d brown jr]

James,


Here's a comparision of perl vs awk in speed against a file this large:


# ls -l match.data
-rw-rw-rw- 1 root sys 46755563 Feb 3 11:05 match.data
# wc match.data
2000000 9866667 46755563 match.data
#

Robin's perl script using "cat FILENAME | perlscript":
real 1:11.2
user 13.7
sys 9.5

My awk using "cat FILENAME | awkscript":
real 1:37.1
user 24.4
sys 16.3

Robin's perl script using "perlscript FILENAME":
real 1:11.2
user 13.7
sys 9.5

NOW notice that the ABOVE tests sent the OUTPUT to the screen, THESE results below are results when the OUTPUT is sent to a FILE:

Robin's perl script:
real 11.4
user 10.9
sys 0.3

My AWK script:
real 20.4
user 19.7
sys 0.3

This proves "perl" is almost twice as fast as "awk".

live free or die
harry