topic Re: Awk help again in Operating System - HP-UX

Awk help again

Belinda Dermody — Thu, 17 Apr 2003 16:48:59 GMT

I have a maillog like below, one entry from a couple hundred thousand per day. Each line starts with a date time stamp. All I want from the log is grab the HeaderTo= and the line before which is HeaderFrom and the line after HeaderSubject

Apr 17 03:11:29 guadeloupe smap[15162]: ACCEPTING MESSAGE - INFORMATION FOLLOWS
Apr 17 03:11:29 guadeloupe smap[15162]: HeaderFrom='"ZZZZZ YYYYYY" acoast.com>'
Apr 17 03:11:29 guadeloupe smap[15162]: HeaderTo=''
Apr 17 03:11:29 guadeloupe smap[15162]: HeaderSubject='Your Perfect Home - New L
istings for 4/17/2003'
Apr 17 03:11:29 guadeloupe smap[15162]: SMTP HostName='smtp.onlinehelp.com'
Apr 17 03:11:29 guadeloupe smap[15162]: SMTP IPAddress='99.9999.999.99'
Apr 17 03:11:29 guadeloupe smap[15162]: SMTP HELO='smtp.onlinehelp.com'
Apr 17 03:11:29 guadeloupe smap[15162]: SMTP MAIL From='ZZZZZZZZ@Seacoast.com'

Re: Awk help again

Pete Randall — Thu, 17 Apr 2003 16:54:17 GMT

James,

How about something like this, with grep, instead:

# print 1 line of context before and after regexp, with line number
# indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

Pete

Re: Awk help again

John Poff — Thu, 17 Apr 2003 17:26:37 GMT

Hi,

Fleet Fingered Pete has given you a solution in sed [I love those sed one liners!]. Here is a little Perl snippet that might work:

perl -ne 'print if /HeaderFrom/ .. /HeaderSubject/' maillog

JP

Re: Awk help again

James R. Ferguson — Thu, 17 Apr 2003 18:01:57 GMT

Hi James:

Yet another (this time in 'awk'):

# WHO=james
# awk -v WHO=$WHO '/HeaderFrom=/ && $0~WHO,/HeaderSubject=/ {print $0}' filename

Regards!

...JRF...

Re: Awk help again

Belinda Dermody — Thu, 17 Apr 2003 18:36:49 GMT

Pete, you almost got it, but the trouble is
I want the three lines that are HeaderTo, HeaderFrom and HeaderSubject.

The key is on HeaderTo for the regexp. Sometimes it is
HeaderTo='

'
or
HeaderTo='address'

any clues

Re: Awk help again

Belinda Dermody — Thu, 17 Apr 2003 18:39:24 GMT

James Ferguson, you are the same. I need to key in on the
HeaderTo and then grab the line before and the line after.

The problem is the HeaderTo could be either
HeaderTo='

'
or
HeaderTo='address'

Re: Awk help again

John Meissner — Thu, 17 Apr 2003 18:43:45 GMT

of course you could always hire an attractive "administrative assistant" and have her pull these lines out by hand while you practice your awk skills :)

Sorry... it's been a looooon week with a 19 hour planned outage this past saturday.... i needed the humor

Re: Awk help again

Belinda Dermody — Thu, 17 Apr 2003 18:47:58 GMT

John I have been practicing my awk skills, I have a habit of trying many different ways and have been working on this since Monday, so I decided to come to the experts, which I guess your field is comedy...

Re: Awk help again

John Meissner — Thu, 17 Apr 2003 18:51:29 GMT

James - I'm not busting on your skills. I have actually tried to figure this type of problem out myslef (stripping out a range in awk & sed) but I don't think i've been that successful in the past. I usually found other ways around this. in order to get a range in sed you can
sed 's/beginning/,/end/g'

Re: Awk help again

John Poff — Thu, 17 Apr 2003 18:52:13 GMT

James,

Is the name you are looking for an e-mail address in the form of somename@somedomain.com, or are you just looking for somename? Does the HeaderTo= always have single quotes around the name?

The Perl example I gave doesn't look for the name at all. I'll try that again.

JP

Re: Awk help again

John Meissner — Thu, 17 Apr 2003 18:53:06 GMT

and out of all my post - I think that was my second attempt at humor.... i'll keep practicing :)

Re: Awk help again

Rodney Hills — Thu, 17 Apr 2003 20:06:17 GMT

How about this-

awk '/HeaderTo=/{ print prev; print $0; getline ; print $0; }/;{prev=$0}' maillog

This will hold the line before in "prev", when HeaderTo is found prev and current line is printed. Then a "getline" reads the next line and prints it.

HTH

-- Rod Hills

Re: Awk help again

James R. Ferguson — Thu, 17 Apr 2003 20:25:23 GMT

Hi (again) James:

OK, try this:

#!/usr/bin/sh
awk -v WHO="HeaderTo=$1" '
{ if ( $0~WHO ) {print LINE;print $0;getline;print $0} else {LINE=$0}
}' filename
exit 0

...pass the argument to match the string that follows "HeaderTo=". Thus if the script were called 'my.sh' do:

# my.sh "'"

...or:

# mysh "james"

...etc., depending on the string following the equal sign.

Regards!

...JRF...

Re: Awk help again

Francisco J. Soler — Thu, 17 Apr 2003 21:18:31 GMT

Hi James,
I attach you a couple of scripts, with the first one the address is embeded in the script itself, in the second one you must type the address in the command line.

- First script.

awk -f file.awk inputfile > outputfile

- Second script

awk -v address="XXXXXXXX@alum\\.colby\\.edu" -f file2.awk inputfile > outputfile

hope this helps.

Frank.

Re: Awk help again

Pete Randall — Thu, 17 Apr 2003 22:26:35 GMT

Sorry James,

While I will always think that the forums are the greatest resource there is, I'm supposed to be starting my vacation (and Momma said NOW!). I would attach all the "Handy One Liners for Sed" that Princess Paula originally posted so you might be able to noodle through - but I'm at home, I can't get to my notes - AND MOMMA SAYS NOW!!!

Pete

Re: Awk help again

John Poff — Fri, 18 Apr 2003 00:01:49 GMT

Pete,

I've got you covered. I keep my sed one liners handy at work. I'm at home now, but I found a recent copy from the author's web page. I've posted it as an attachment.

JP

P.S. I understand what it means when Momma says NOW! Enjoy your vacation. :)

Re: Awk help again

Pete Randall — Fri, 18 Apr 2003 00:20:02 GMT

Thanks, John!

I knew someone would pick up the slack. I'll send you a post card!

Pete

Re: Awk help again

Belinda Dermody — Fri, 18 Apr 2003 11:52:52 GMT

Thanks to all you guy's, I haven't tried all the new responses yet, but that is why I have always insisted to my manager to make sure whatever support we have for our HP system that I have the forum option available. I very seldom call any longer, The response from the site (might be slow getting to it from the internet once in a while) is the greatest and the best and the diversacation of answers and abilities are the best and I have responded that to the leaders anumber of times.

We support online communities for Universities and Orgs and one of our perks is a email forwarding capabilities and of course the upper management like daily numbers and stats.
the To address could have ' ' around the address or < > or '< >' so I would just like to grab the address within the special chars.

Re: Awk help again

James R. Ferguson — Fri, 18 Apr 2003 12:18:56 GMT

Hi (again) James:

Based on your last comments, if you could find the lines you want regardless of how they are bounded, using the script I supplied, thusly:

# my.sh "'*<*James>*'*"

...would return matches for:

HeaderTo=James
HeaderTo=
HeaderTo=''
HeaderTo='James'

Regards!

...JRF...

Re: Awk help again

James R. Ferguson — Fri, 18 Apr 2003 13:14:17 GMT

Hi (again) James:

Since you're filtering mail logs, perhaps you like to extract (match) names case-insensitively, so that "James" is the same as "james". If so try this script with your data:

#!/usr/bin/sh
typeset -l WHO="HeaderTo='*<*${1}>*'*"
awk -v WHO=$WHO '
{ if (tolower($0)~WHO) {print LINE;print $0;getline;print $0} else {LINE=$0}
}' filename
exit 0

...now to extract any variation of "jAmEs":

./my.sh jAmEs

Note that you pass one argument representing the name to match.

Regards!

...JRF...