bash script to check for a specific string in the log file and display by hourly

[shankar!]

Hi All,

Good day,

I am writing a BASH script in linux, the purpose of this script is to check for a specific string (ex, InitialDP) in an application log file and if found any such string, then it should be displayed as a count on hourly basis. I wrote this,

grep "`date +'%b %e %H'`" cap3dgw.log | grep -i InitialDP > temp

echo "Display all lines that containig the string"

echo

cat temp

echo

echo "Counting the lines that contains the string, `cat temp | wc -l`"

But the above script will display results only for that current hour. Say If I ran that script now, 5:08 PM IST, I get all results of 5:08 hour only. I want the output count of each hour in the log file if such string found.
ex: 5:00 hour --> counting the lines that contains the word is, 15

6:00 hour --> counting the lines that contains the word is, 3

7:00 hour --> no count as no such word found..

so on and on..

Can someone please help me on this

thanks
-asam

[Suraj K Sankari]

Hi,

Put your script into crontab and set to run every hour.

Suraj

[shankar!]

Thanks Suraj,

Is there any other possible way to do this? Like if I run the script now, it should get all the counts for each hour and display it.

The reason why I am doing this is, the log file name will be different every time( like, cap3dg.log, cap4dg.log, cap5dg.log etc).

Thanks

[Mel Burslan]

If I am understanding you correctly, you want to run this script once and want to get the output for every hour in this file, line-by-line. If this is your desire, you need to give an example of few lines (5-10) from your log file, so that a time interval filter could be suggested. Without know how time gets imprinted into your log file, it is almost impossible to suggest anything

[James R. Ferguson]

Hi Asam:

You could do something like this:

# cat ./look
#!/usr/bin/sh
typeset FILE=$1
typeset MONDD=$(date +'%b %e')
typeset MAXHR=$(date +'%H')
typeset -i HH=0
typeset PLACE
typeset MATCH
typeset -i COUNT

while (( HH <= MAXHR ))
do
[ ${HH} -lt 10 ] && PLACE=0 || PLACE="" ];
if [ ${HH} -lt 10 ]; then
PLACE=0
else
PLACE="+"
fi
MATCH="${MONDD} ${PLACE}${HH}"
COUNT=$(grep -c "${MATCH}" ${FILE})
echo "${MATCH} = ${COUNT}"
(( HH=HH+1 ))
done
exit

...run as:

# ./look file

Notice that 'grep' can count matches without having to spawn a new process like 'wc' to do that.

Regards!

...JRF...

[shankar!]

yes, correct. I want to display the output for every hour line-by-line ( later we can wc -l to see count for every hour). some lines from log file are,

T<1>Fri Mar 20 13:18:50 2009 : [handleTCAPMessage] Recieved InitialDP < 4726="">
T<1>Fri Mar 20 13:18:50 2009 : [handleTCAPMessage] Before DecodingL_pComp->parameter->actual_length 71
T<1>Fri Mar 20 13:18:50 2009 : [handleTCAPMessage] IDPDecoding Success < 4777="">
T<1>Fri Mar 20 13:18:50 2009 : Location Number is (MSC addr )is : < 3767=""> 50378629012
T<1>Fri Mar 20 13:18:50 2009 : Sending ETC Message on
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2352 | IP Routing Address = c
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2352 | IP Routing Address = 3
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2352 | IP Routing Address = 10
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2352 | IP Routing Address = 99
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2352 | IP Routing Address = 39
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2396 | Correlation Id = 6
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2396 | Correlation Id = 0
T<1>Fri Mar 20 13:18:50 2009 : buildEstablishTemporaryConnection | 2396 | Correlation Id = 0
T<1>Fri Mar 20 13:18:50 2009 : CAP3_getEncodingLength Successful = 14
T<1>Fri Mar 20 13:18:50 2009 : CAP3_encoding successful
T<1>Fri Mar 20 13:18:50 2009 : TCX Put component is sucessfull

[Mel Burslan · Accepted solution]

Okay, assuming the file gets regenerated or renamed daily (s there is not a double occurrence of any hour, you can employ something like this

grep "My Search String" my_log_file > /tmp/skimmed_log
TEMPFILE=/tmp/logparser_tmp
FILE=/tmp/skimmed_log
for hour in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
do
cat $FILE | while read line
do
timeh=$(echo ${line}|awk{'print $4'}|cut -d: -f1)
hourcount=0
if [ "${timeh}" = "${hour}" ]
then
(( hourcount=${hourcount}+1 ))
fi
done # end of while loop
echo "${hour} hour --> counting the lines that contains the word is, ${hourcount}"
done # end of for loop

if your timestamp uses single digits for hours 0 thru 9, you need to modify the parameters for the "for" loop as it is designed to do string comparison and "0" will not be equal to "00" in its current form.

Hope this helps and be careful for the syntax and typographical errors I might have done as I have no way of testing this code.

[James R. Ferguson]

Hi (again) Asam:

Sorry, I focused on showing you how you might count the number of hourly events for a particular day without regard to what the specific "event" was.

If we assume that your timestamp occurs before the string of interest we can summarize the counts you want using the same approach I first took. At least, this will point you in a direction.

If all you want to do is 'grep' for lines in a particular hour and output those, do:

# grep -E "May 15 09.+InitialDP" file

This would show you all the lines in 'file' with the string "InitialDP" somewhere after the sting "May 15 09".

For summarizing the counts by hour we might use:

# ./look
#!/usr/bin/sh
typeset WHAT=$1
typeset FILE=$2
typeset MONDD=$(date +'%b %e')
typeset MAXHR=$(date +'%H')
typeset -i HH=0
typeset PLACE
typeset TOKEN
typeset -i COUNT

while (( HH <= MAXHR ))
do
[ ${HH} -lt 10 ] && PLACE=0 || PLACE="" ];
TOKEN="${MONDD} ${PLACE}${HH}.+${WHAT}"
COUNT=$(grep -E -c "${TOKEN}" ${FILE})
echo "${MONDD} ${PLACE}${HH} = ${COUNT}"
(( HH=HH+1 ))
done

...run as:

# ./look InitialDP file

That is, pass the string to be matched and the name of the file to search.

Regards!

...JRF...

[Mel Burslan]

sorry, the line
FILE=/tmp/skimmed_log
should have been removed but I forgot
(no points for this post please)

[shankar!]

Hi James,

The script you gave me working kool, thanks. The only thing is it checks and display the results of current date and time. I have a log file of last weeks date and time. If i use your scripts to run it, it does checks for today's date and hour only

ie
May 15 00 = 0
May 15 01 = 0
May 15 02 = 0
May 15 03 = 0
May 15 04 = 0
May 15 05 = 0
May 15 06 = 0
May 15 07 = 0

Is there any way we can get results for any date/time (like any day or month) of each hours?

Mel: I did not tried your solution as James solution was working fine. thanks to your time

Thanks for all help.

[Dennis Handly]

>Is there any way we can get results for any date/time (like any day or month) of each hours?

Basically you want a summary for each hour for any date?

awk '
BEGIN { monthp=""; dayp=0; hourp=0; count=0 }
/InitialDP/ {
# get current time
month = $2
day = $3
hour = substr($4, 1, 2) # assuming a rational locale
if (month == monthp && day == dayp && hour == hourp)
count++
else {
if (monthp != "")
print monthp, dayp, hourp, "=", count
monthp = month
dayp = day
hourp = hour
count = 1
}
}
END { print monthp, dayp, hourp, "=", count }
' $*

[Dennis Handly]

My awk script above doesn't report hours with zero occurrences.

[shankar!]

Thank you Dennis, that helped us a lot...

And thanks all, thanks to the great community...

Honestly
-asam