topic Re: Awk question? in Operating System - HP-UX

Awk question?

Allanm — Mon, 05 Oct 2009 17:48:14 GMT

I have the following entry in my web logs -

1.1.1.1 - - [17/Sep/2009:03:14:50 -0700] "GET /c14/v23.129/011157032/getlocalstring?template=calendar HTTP/1.1" 200 8941 "https://URL/c14/v28.129/011157032/reports/execute?rptid=011157032-VEND_BAL_DET-view-1253182486693" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" URL 80 - - - 0 "-" "1.1.1.1.232161235455457383"

I need to get the IP address(1.1.1.1-this can vary) and account id(011157032- this can vary) from each of the entries which are of this pattern within the web logs.

How do I get them?

Thanks,
Allan

Re: Awk question?

Hein van den Heuvel — Mon, 05 Oct 2009 18:12:05 GMT

You may want to consider the PERL regex over AWK.

Here is something that works with the example in a file 'test'.

$ perl -ne 'print qq(ip=$1 acc=$2\n) if m%^(\S+)\s.*?/(\d+)/\w+%' test
ip=1.1.1.1 acc=011157032

Now the string 1.1.1.1 appears twice. Which one is authorative?

I picked the fist one as being all the non-space starting at the begin of the line.

The Account number appears 3 times? Which one will always be there, which one has the most predictable 'surroundings'?

I picked the first one between two slashes and being followed by a word (getlocalstring in the example) followed by a ?

For a better solution, please provide a better problem description!

Regards,
Hein

Re: Awk question?

James R. Ferguson — Mon, 05 Oct 2009 18:30:33 GMT

Hi Allan:

Perhaps:

# perl -nle '/((?:\d+\.?)+).+rptid=(\d+)/ and print join ":",$1,$2' file

1.1.1.1:011157032

Regards!

...JRF...

Re: Awk question?

Matti_Kurkela — Mon, 05 Oct 2009 18:35:39 GMT

While this is certainly possible with awk, this can be solved with a humble "cut" too.

"sed" would also work, but sed solutions all too easily become painfully Write-Only :-)

Is the account ID always prefixed by "/c14/v23.129/" or can this vary too? Since this looks like a version number, I'll assume the numbers might vary, but the number of path elements will stay the same.

If the prefix can vary, is it guaranteed that there request URI will have no spaces and a total of three slashes before the account ID? And that the account ID is always terminated with a slash?

If so, then pipe your log to this little script: (i.e. execute with "scriptname < weblog.log")

#!/bin/sh

while read logline
do
ipaddr=$(echo "$logline" | cut -d " " -f 1
requestURI=$(echo "$logline" | cut -d " " -f 7)
accountID=$(echo "$requestURI" | cut -d / -f 4)
echo "Account $accountID from IP $ipaddr"
done

MK

Re: Awk question?

Allanm — Mon, 05 Oct 2009 20:18:14 GMT

Thanks Hein!

Works even with \s.* while omitting the ?.

Thanks,
Allan

Re: Awk question?

Allanm — Mon, 05 Oct 2009 20:19:06 GMT

JRF , can explain the regex ...

Thanks,
Allan.

Re: Awk question?

James R. Ferguson — Mon, 05 Oct 2009 20:45:30 GMT

Hi (again) Allan"

> JRF , can explain the regex ...

# perl -nle '/((?:\d+\.?)+).+rptid=(\d+)/ and print join ":",$1,$2' file

Match and capture one or more digits (\d+) followed by an optional dot character (\.); repeating one more times; followed by any number of characters and the pattern "rptid=". The inner parentheses are non-capturing (?:) ones since we don't need them for anything other than a group repetition. Thus the address is captured in $1. Following the "rptid=" pattern we capture one or more digits that follow in $2.

If the whole pattern is satisfied, we print the pieces that were stored in $1 and $2, joining them together with a colon character.

Regards!

...JRF...

Re: Awk question?

Dennis Handly — Tue, 06 Oct 2009 13:46:59 GMT

This looks like a continuation of your other threads:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1375401

Re: Awk question?

James R. Ferguson — Tue, 06 Oct 2009 14:07:35 GMT

Hi (again) Allan:

So Dennis was correct, your requirements were better defined in you earlier thread :-) namely "The account id will always be the 3rd entry from /(slash) - with t15 being server-number, v33.109- code version and 670612718 being the account id."

Given this, I would offer:

# perl -nle 'm{^((?:\d+\.?){4}).+"GET\s+/.+?/.+?/(\d+)} and print join ":",$1,$2'

This also anchors the the isolation of the IPaddress to the beginning of each line.

Regards!

...JRF...

Re: Awk question?

user57 — Wed, 07 Oct 2009 06:09:35 GMT

For the given data:

$ awk -F '[ - / ]' '{print $1,$12}' wlog
1.1.1.1 011157032

Re: Awk question?

user57 — Wed, 07 Oct 2009 11:18:13 GMT

You did want an awk solution. For a 2 line file:

awk -F \" '
{
split($1,IP,"-")
split($2,ID,"/")
print IP[1],ID[4]
}' wlog

./web.awk
1.1.1.1 011157032
1.1.1.2 011157033