need ideas on reporting from syslog.log

[Chris Fadrowski]

I would appreciate any ideas you may have regarding how to pull specific information out of syslog.log and send an email page to a pager..

so far, i can grep for any issues using

grep -Fi -e err -e warn -e crit /var/adm/syslog/syslog.log | /usr/bin/mailx -s "warning" email@email.com

however, i only want to grep new items and not the entire file every time i cron this command. I am not good enough to script this using 'awk', so is there any other way to do it? any ideas?

[Pete Randall]

Chris,

Grep them into a separate file with

grep -Fi -e err -e warn -e crit /var/adm/syslog/syslog.log > warn.out

This will re-create the file with just the latest messages each time it's run and you can check the file to decide whether the warning should be issued.

Pete

[Chris Wilshaw]

How about this.

Create a file containing all errors grepped from syslog.log to the current time (your base file).

at whatever interval you require, grep the list of errors again from syslog.log into another file.

Run a diff or comm against the 2 files, and if any differences are reported, send your alert.

Then copy the newly created file over the base file, so that the next tim you run, you are comparing the differences between "now" and your last run.

[Dietmar Konermann]

What about not using cron, but a background job like this (only a rough explample to show the idea):

tail -f /var/adm/syslog/syslog.log |
while read line; do
echo "$line" | grep -Fi -e err -e warn -e crit | mailx -s warning email@email.com
done

Best regards...
Dietmar.

[David_246 · Accepted solution]

Hi,

This is funny. I am currently implementing the tool logsurfer to do exactly the same as you want.
This is easy to use but needs a little finetuning, but I can help you with that.

You can specify an offset (wc -l) while starting logsurfer. If you send me your e-mail adress I will send you all the info and source.

Regs David
e-mail david.vandergeer@axa.nl

[Chris Fadrowski]

but how could i get it to run automatically lets say twice a day. I am trying to automate it as much as possible.

[Pete Randall]

Chris,

Are you familiar with cron? Do a man on crontab. You could set your first grep to run twice a day producing the warn.out file, then set another job (in cron) to check that warn.out file 5 or 10 minutes after the first job creates it.

Pete

[Pete Randall]

Cron entries would look something like this:

00 6,18 * * * checklog.sh > warn.out

05 6,18 * * * checkwarn.sh



Pete

[David_246]

Just to make it available to anyone :
Hi Chris,

You can just "gzip -d logs.tar.gz; cd /usr/local; tar xvf logs.tar"
This unpacks a directory called logsurfer in your /usr/local

The configuration script we use is /usr/local/logsurfer/etc/conf.syslog.

You start it using : /usr/local/logsurfer/bin/logsurfer -c
So : /usr/local/logsurfer/bin/logsurfer -c /usr/local/logsurfer/etc/conf.syslog /var/adm/syslog/syslog.log

The configuration script just uses regular expressions, they might look awfull sometimes, but in fact they are very easy. If you take five minutes you will understand almost all of it. And of course, for questions you can send me an e-mail.

The start-up script for the /sbin/init.d - directory is attacht'ed as "Naamloos"
Don't forget to take the e-mail adres "logsurfer" in you /etc/mail/aliases and run an "newalias" afterwards.

I have three sites with info, one is in German, I don't know if you speek german. Lucky enough I do a little, the otherone is just english so should be no problem.
http://www.hwk-duesseldorf.de/linuxkurs/redhat-37.html
http://www.cert.org/security-improvement/implementations/i042.02.html
http://www.cert.dfn.de/eng/logsurf/

Please don't hessitate to ask if you need.


Regs David

/sbin/init.d/logsurfer
#!/bin/sh
#

pid=`/usr/bin/cat -s /usr/local/logsurfer/logsurfer.pid`
user=`/usr/bin/id|/usr/bin/sed 's/.*(\(.*\)) .*/\1/'`

# test if listed process is still active
if [ "${pid}" != "" ]; then
pid=`/usr/bin/ps -p ${pid} | /usr/bin/tail -1 | awk '{print $1}'`
fi

# handle the option specified
case $1 in
start_msg)
echo "Start Logsurfer"
;;

stop_msg)
echo "Stop Logsurfer"
;;

start)
# start the logsurfer process
if [ "${pid}" != "" ]; then
/usr/bin/echo "\nLogsurfer System is already initialized."
exit 0
fi
if [ -f /usr/local/logsurfer/bin/logsurfer -a -f /usr/local/logsurfer/etc/conf.syslog ];
then /usr/local/logsurfer/bin/logsurfer -c /usr/local/logsurfer/etc/conf.syslog -l `/usr/bin/wc -l < /var/adm/syslog/syslog.log` -d /usr/local/logsurfer/etc/dumpfile -p /usr/local/logsurfer/logsurfer.pid -f /var/adm/syslog/syslog.log &

else
/usr/bin/echo "\nNo Logsurfer program or configuration found."
fi
;;

newstart)
# restart the logsurfer process
if [ "${pid}" = "" ]; then
/usr/bin/echo "\nNo Logsurfer System initialized. \n"
exit 0
fi

# stop the actual process
/sbin/init.d/logsurfer stop
# wait a while
/usr/bin/sleep 5
# start a new process
/sbin/init.d/logsurfer start
;;

stop)
# stop all logsurfer processes
if [ "${pid}" = "" ]; then
/usr/bin/echo "\nNo Logsurfer System initialized."
exit 0
fi
/usr/bin/kill ${pid}
/usr/bin/rm /usr/local/logsurfer/logsurfer.pid
;;

*)
/usr/bin/echo "Usage: /sbin/init.d/logsurfer { start|newstart|stop }"
;;

esac

[David_246]

Oops, forgotten to explain in the previous message. When starting logsurfer it might be very usefull to give the offset (-l option) too :
/usr/local/logsurfer/bin/logsurfer -c /usr/local/logsurfer/etc/conf.syslog -l `/usr/bin/wc -l < /var/adm/syslog/syslog.log`
-d /usr/local/logsurfer/etc/dumpfile -p /usr/local/logsurfer/logsurfer.pid -f /var/adm/syslog/syslog.log &

Else your -email adres will be bombed with messages :(

Regs David

[linuxfan]

Hi Chris,

You may also want to check Logsentry (used to be called logcheck). Its a pretty nifty tool.

The main web site is http://psionic.com

You can download logsentry from
http://psionic.com/downloads/logsentry-1.1.1.tar.gz

-HTH
Ramesh

[Carlo Henrico_1]

I see you have got some good answers but to address you original question (...not the entire file every time...) why not try "dmesg -". Doing a dmesg shows all problems in your syslog.log, however adding the "-" to it only shows the new ones! The first time you do it, it still shows everything but thereafter only the new stuff.

Good luck

Carlo

[Rory R Hammond]

How about looking at this differently.

syslog is a product of the syslogd
edit /etc/syslog.conf
have syslogd also write interesting errors to a syspage.log file.
you can then grep page qualty informatin from the syspage.log, immediatly zero out the file so that you can page your self when new stuff arrives. When paged you can look in syslog.log for the exact messages and circumstances. Rememember ZERO out the syspage.log file don't remove it.

Rory