need help on a script

[Jan Shu]

Hi All,

Greetings. Can you help me with a script? I created a simple script to grep the last 20 lines in syslog.log if there is any hardware critical (5) error. And then send email pages to my pager. This cron job runs every 15 min. The problem is that it keeps sending pages every 15 min until I stop the cronjob. How to modify the script so it sends out ONLY ONE alert page.

Thank you for your help in advance.

***** here is my script *****
tail -20 /var/adm/syslog/syslog.log | grep "CRITICAL (5)"
if [ $? -eq 0 ]
then
tail -20 /var/adm/syslog/syslog.log | grep "CRITICAL (5)" | mailx -s "HPUX server03 HARDWARE WARNING" my_email_address
fi

Best Regards,
Jan Shu

[Geoff Wild]

The problem you have in general is, if there is a critical alert - and then nothing else for 15 minutes - you will get paged on the same alert....as you are tailing the file...

If you really want to do some monitoring - and can't afford OVO - then check out BigBrother:

http://www.bb4.org/

Rgds...Geoff

[Jan Shu]

Hi Geoff,

We do have big brother and HP SIM software, but they weren't setup to monitor my HPUX servers yet. In the meantime, I am looking for a simple script to send hardware alert pages, if it is possible.

Thanks again.

Jan

[Gary L. Paveza, Jr.]

I use the following script. Basically, it eliminates the need to watch for X number of lines because it does a diff on the syslog file. You can add any errors you want just by adding to the array listed.

One drawback is that once you get an error, the next time you reboot, you'll be notified again (because the copy of syslog has it, but the current syslog doesn't). This could be eliminated by looking for entries just in the current syslog, but not in the copy of syslog, but I haven't gotten around to coding that.

#!/usr/bin/ksh
#
# @(#) Check syslog for errors
# @(#) $Revision 1.0
# @(#) Author: Gary Paveza
# @(#) Created 2003/09/16
# @(#) Last Modified: 2003/09/16
#

CP=/usr/bin/cp
DIFF=/usr/bin/diff
GREP=/usr/bin/grep
MAILX=/usr/bin/mailx
RM=/usr/bin/rm
SORT=/usr/bin/sort
UNAME=$(/usr/bin/uname -n)

DIFF_SYSLOG=/var/tmp/diff_syslog.log
EMAIL_LIST=""
ERROR_LOG=/var/tmp/errorlog.$$
SYSLOG=/var/adm/syslog/syslog.log
STATIC_SYSLOG=/var/tmp/syslog.log

MONITOR_ERROR[0]="cmcld: lan1 failed"
MONITOR_ERROR[1]="cmcld: lan4 failed"
MONITOR_ERROR[2]="cmcld: lan1 switched to lan6"
MONITOR_ERROR[3]="cmcld: lan6 switched to lan1"
MONITOR_ERROR[4]="cmcld: Subnet XXXXX switched from lan1 to lan6"
MONITOR_ERROR[5]="cmcld: lan1 recovered"
MONITOR_ERROR[6]="cmcld: lan6 recovered"
MONITOR_ERROR[7]="cmcld: HB connection to 192.168.1.2 not responding, closing"
MONITOR_ERROR[8]="cmcld: GS connection to 192.168.1.2 not responding, closing"
MONITOR_ERROR[9]="cmcld: HB connection to 192.168.1.2 is responding"
MONITOR_ERROR[10]="cmcld: GS connection to 192.168.1.2 is responding"
MONITOR_ERROR[11]="cmcld: HB connection to 192.168.1.1 not responding, closing"
MONITOR_ERROR[12]="cmcld: GS connection to 192.168.1.1 not responding, closing"
MONITOR_ERROR[13]="cmcld: HB connection to 192.168.1.1 is responding"
MONITOR_ERROR[14]="cmcld: GS connection to 192.168.1.1 is responding"
MONITOR_ERROR[15]="vmunix: LVM: Performed a switch for"

#
# Check to see if the static syslog file is present. If not, create it.
#
if [ ! -f ${STATIC_SYSLOG} ]
then
${CP} ${SYSLOG} ${STATIC_SYSLOG}
fi

#
# Create diff syslog (compare syslog with static syslog)
#
${DIFF} ${SYSLOG} ${STATIC_SYSLOG} > ${DIFF_SYSLOG}

#
# Scan through diff file looking for errors which are defined in the
# MONITOR_ERROR errors.
#

INDEX=0
while [ ${INDEX} -lt ${#MONITOR_ERROR[*]} ]
do
${GREP} "${MONITOR_ERROR[${INDEX}]}" ${DIFF_SYSLOG} >> ${ERROR_LOG}
INDEX=$(expr ${INDEX} + 1)
done

#
# Email out any errors found
#
if [ -s ${ERROR_LOG} ]
then
${SORT} ${ERROR_LOG} | ${MAILX} -s "Syslog errors for ${UNAME}" ${EMAIL_LIST}
fi

#
# Reset environment
#
${RM} ${ERROR_LOG}
${CP} ${SYSLOG} ${STATIC_SYSLOG}

[Jan Shu]

Hi Gary,
Thanks for the reply. I will need to try that script on a dev server first.

Jan

[Geoff Wild]

Also tak a look at this thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=401290

Tim has a way to do it so you only see today's date...

Should be able to modify it so it only looks for anything after the last time it was run...


Rgds...Geoff

[Gary L. Paveza, Jr.]

Jan,

make sure you modify it for your site. Things like lan1, etc are system specific.

[Jan Shu]

Hi Geoff,
Thanks for the info. I looked at Tim's script, but it still needs some work-around.
Regards,
Jan

[Bill Hassell · Accepted solution]

Pager alerts are always a problem. Whether yuo're monitoring disk space or syslog, you need something that tracks what has already been done. For a script I wrote to monitor disk space, I had to create a config file to set the warning limit (like 90% full or more), then an increment amount (2% for example). What the script would do is send one message at 90% or more, then if the current value changed 2% or more, send another message. If the amount doesn't change at the next check, no messdage is sent.

To do this, you need to track what has already been sent in a file, then check to see if it has already been sent. For syslog, it's actually easier because it is sequential and the timestamp continues to increase. Monitoring the last NN lines of syslog isn't a good idea as a single disk error can create 10-25 lines of messages. Instead, you can append your own message to the end of syslog with the logger command (the -t option is useful). That signals the end of the last scan. The next time you run the script, look for the last marker in syslog and everything after that point needs to be scanned.

[Jan Shu]

Hi Bill,

Though each of the inputs above was very helpful, I think your logic seems to be easier to understand for me. Can you please give me an example on "-t" option. I am going to add that to my script.

Many Thanks.

Regards,
Jan

[Bill Hassell]

logger -t "SomeUniqueText" -p user.warn "more text"

This will insert a line at the end of syslog. Technically, you don't need the -t tag as your unique_text can be the message portion:

logger -p user.warn "unique_text"

Now your script will search for this unique_text using grep -n which produces line numbers for each match:

grep -n "unique_text" /var/adm/syslog/syslog.log | tail -1

Now you have the last line number in syslog with your matching text. Now compare the actual length of syslog with your last line number:

LASTSCAN=$(grep -n "unique_text" /var/adm/syslog/syslog.log | tail -1)

LOGLEN=$(wc -l /var/adm/syslog/syslog.log | awk '{print $1}')

if [ $LASTSCAN -lt $LOGLEN ]
then
... scan for errors and page if needed
logger user.warn "unique_test"
fi


So the script fragment checks if anything was added after the last scan. If not, do nothing. If true, scan from $LASTSCAN to the end of the file and page as needed. Then add another logger entry to establish a new end-of-scan point for the next script run.

[Jan Shu]

Hi Bill,

Thank you for your help very much. I will update my script according to your information.

Best Regards,
Jan Shu

[Jan Shu]

THANK YOU ALL FOR THE INPUTS.
HAPPY NEW YEAR TO YOU AND YOUR FAMILIES.