topic Re: shell script for log parsing in Operating System - Linux

shell script for log parsing

Shivkumar — Sun, 01 Jan 2006 00:12:10 GMT

Dear Sir,

I want to find lines containing error message "500 Server Error: 20-0002" in log file.

A sample entry in the log containing the error message is shown below:

[15/Dec/2005:10:17:47-28247-160-0] Process - Exiting with HTTP 500 Server Error: 20-0002

I want to parse this log for the error "500 Server Error: 20-0002" between time period 10:00:00 to 18:00:14
for 2 dates (1) between 10/Dec/2005 to 15/Dec/2005 (2) only on 15/Dec/2005

Can someone suggest a shell script ?

Regards,
Shiv

Re: shell script for log parsing

Steven E. Protter — Sun, 01 Jan 2006 06:13:37 GMT

hits=$(grep "500 Server Error: 20-0002" /var/adm/syslog.log | wc -l)

if [ $hits -ge 1 ]
then
# insert notify code here.
else
echo all is well in systemland
fi

Adjust the file being grepped to your needs.

SEP

Re: shell script for log parsing

john korterman — Sun, 01 Jan 2006 13:10:52 GMT

Hi,

one simple approach would be to use awk to search for the lines in between to strings, e.g.:

$ awk '/\[10\/Dec\/2005:10/,/\[10\/Dec\/2005:18/' your_logfile

the awk pattern is basically
/from/,/to/
the backslashes are for escaping the slashes in the search strings.

An awk statement can be combined with grep in a script, e.g.:

#!/usr/bin/sh
awk '/\[10\/Dec\/2005:10/,/\[10\/Dec\/2005:18/' $1 | while read line
do
grep "Process - Exiting with HTTP 500 Server Error: 20-0002"
done

which you can run using your infile as $1.
However, both the "from" and "to" string *must* exist in your_logfile

I suggest you do it one day at a time.

regards,
John K.

Re: shell script for log parsing

James R. Ferguson — Sun, 01 Jan 2006 13:49:00 GMT

Hi Shiv:

This becomes easy in perl. Although this script could be improved, it provides a general guideline. It's easy to compare date/time ranges if you translate them into epoch seconds (the number of seconds since January 1, 1970). That's what I do here. Since you presented your dates in the Europenan format, I honored that too.

You an change the $pattern and/or $firstdt, $firsttm, $lastdt, $lasttm values to your needs. Since this is a quick script, I didn't choose to pass them as arguments.

# cat logpeek

#!/usr/bin/perl
use strict;
use warnings;
use Date::Calc qw( Decode_Date_EU Date_to_Time );
my ($year, $month, $day, $hour, $min, $sec);
my ($date, $time, $first, $last);
my @time;
my $pattern = "500 Server Error: 20-0002";
my $firstdt = "10 Dec 2005";
my $firsttm = "10:00:00";
my $lastdt = "15 Dec 2005";
my $lasttm = "18:00:14";

die unless (($year, $month, $day) = Decode_Date_EU($firstdt));
@time = split( /:/, $firsttm);
$first = Date_to_Time($year, $month, $day, $time[0], $time[1], $time[2]);

die unless (($year, $month, $day) = Decode_Date_EU($lastdt));
@time = split( /:/, $lasttm);
$last = Date_to_Time($year, $month, $day, $time[0], $time[1], $time[2]);

while (<>) {
next unless m/$pattern/i;
next unless m/\[(\d+.+\d{4}):(\d+:\d+:\d+)/;

$date = $1;
@time = split( /:/, $2);
($hour, $min, $sec) = (@time) [0..2];

if (($year, $month, $day) = Decode_Date_EU($date)) {
$time = Date_to_Time($year, $month, $day, $hour, $min, $sec);
next if ($time < $first or $time > $last);
print $_;
}
}
1;
#_jrf_

Run the script, passing it the name of your log file, as for example:

# ./logpeek logfile

Regards!

...JRF...

Re: shell script for log parsing

Shivkumar — Sun, 01 Jan 2006 17:13:50 GMT

I wish a very happy and prosperous new year to forum moderators and members of this forum. This is really a great forum and i have never come across a forum on unix like this. I wanted to learn unix from gurus and experts. I am grateful to hp and its great forum members here.

warm regards to all,
Shiv

Re: shell script for log parsing

Shivkumar — Sun, 01 Jan 2006 18:07:01 GMT

How to verify whether perl would be available on my hpux box ? Does it come by default on all hpux servers ?

Re: shell script for log parsing

James R. Ferguson — Sun, 01 Jan 2006 18:33:54 GMT

Hi Shiv:

It is available for installation in all recent releases. You can verify that you (*should*) have it with:

# swlist -l product perl

If it's not installed you can get it here:

http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=PERL

...or an even more recent version, here, thanks to Merijn (Procura):

http://mirrors.develooper.com/hpux/

Regards!

...JRF...

Re: shell script for log parsing

James R. Ferguson — Sun, 01 Jan 2006 18:47:41 GMT

Hi (again) Shiv:

I should add this information. If you do:

# whereis perl

You may get back something like:

perl: /usr/bin/perl /usr/contrib/bin/perl /opt/perl/bin/perl /opt/perl_64/bin/pe
rl /opt/perl/man/man1/perl.1 /opt/perl_64/man/man1/perl.1

You need to assess the version of perl that is thus available. I have soft-linked '/opt/perl/bin/perl' and '/usr/bin/perl'. For instance:

# /opt/perl/bin/perl -v

...returns (in part):

"This is perl, v5.8.2 ..."

This is a fairly current version. My point is that if I do:

# /usr/contrib/bin/perl -v

...I see:

This is perl, version 5.005_02 ..."

This is the very *old* version that will exist on your system regardless of whether or not you installed a current version of perl. You want to *at least* use a 5.8.x version.

Regards!

...JRF...

Re: shell script for log parsing

Arunvijai_4 — Sun, 01 Jan 2006 22:45:57 GMT

Hi Shiv,

How to verify whether perl would be available on my hpux box ? Does it come by default on all hpux servers ?

To answer your question, perl comes default on 11.11 and 11.23. You can find the version of perl by # perl -v or # swlist |grep -i perl

If you want to download perl, http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=PERL

Or from Procura's site,

http://mirrors.develooper.com/hpux/

-Arun

Re: shell script for log parsing

Muthukumar_5 — Mon, 02 Jan 2006 01:24:20 GMT

Use this:

# cat test.sh
# Change to your log file location
LOGFILE=test.log

awk '/500 Server Error: 20-0002/ { print $0; }' ${LOGFILE} | while read line
do

echo "On Dec 15th betweentime period 10:00:00 to 18:00:14"
echo

date=$(echo ${line} | awk -F"/" '{ print $1; }' | tr -d '[')

if [[ $date -eq 15 ]]
then

result=$(echo ${line} | awk -F: '{ if ( ($2 >= 10 || $2 <= 18) && ($3 >= 0 || $3 <=0) && ($4 <=14 || $4 >= 0) ) { print "
1"; }}');

if [[ $result -eq 1 ]]
then

echo $line;
fi

echo
echo "Dec 10th to Dec 15th betweentime period 10:00:00 to 18:00:14"
echo

date=$(echo ${line} | awk -F"/" '{ print $1; }' | tr -d '[')

if [[ $date -ge 10 && $date -le 15 ]]
then

result=$(echo ${line} | awk -F"-" '{ print $1; }' | awk -F":" '{ if ( $2 >= 10 && $2 <= 18 && $3 >= 0 && $3 <=0 && $4 <=1
4 && $4 >= 0 ) { print "1"; }}');
if [[ $result -eq 1 ]]
then
echo $line;
fi
fi

fi
done

-Muthu