Need Scripting Help - Script to send email if server goes down

[Laurie A. Krumrey]

Hi All,

I have a (borne shell) script in cron that runs once a day and it monitors my global syslog. All my servers send a copy of their
syslog to this one server call GOAT.

It works great, I get an email and a pager email. The problem is I want it to run in cron
every 10 minutes. Well, I did that and I kept
getting emails telling me the same thing.

So I need to have an output file i.e. "sentmail" so it will not send me the same
email every 10 minutes.

Here's my script, I need help creating and
testing the sentemail file. I'm not sure how
the logic works:

!/usr/bin/ksh
# #
DATE=`date "+%b %d"`
HOST=`uname -n`

#ADMIN_PAGER=9131234420@messaging.sprintpcs.com
ADMIN_EMAIL=lakrumrey@statestreetkc.com

# Check to see if any of the servers are going down (shutdown or reboot)

if [ `grep -i "going down" /var/adm/syslog/syslog.log | wc -l` > 0 ]

then
grep "$DATE" /var/adm/syslog/syslog.log | grep "Going down" | mailx -s "Server Going Down" $ADMIN_EMAIL

fi

[Sachin Patel · Accepted solution]

Hi Laurie,
We run script every 10 minutes to check for n servers and if anyone is not reachable it sends me email.
Here is the loop without going through variable initilization.

for MACHINE in $MACHINES; do
RESULT=`$PING $MACHINE $BYTES $COUNT | grep "packet loss" | sed 's/%//g' | awk '
{print $7}'`
if [ "$RESULT" = "" ]; then
echo "Unexpected ping response from $MACHINE"
elif [ $RESULT -ge 100 ]; then
WASDOWN=`grep "^$MACHINE:" $OLDLIST | wc -l`
if [ $WASDOWN -gt 0 ]; then
grep "^$MACHINE:" $OLDLIST >> $NEWLIST
else
SENDMAIL=1
echo "$MACHINE: $DATE" >> $NEWLIST
echo "$MACHINE NOT RESPONDING" >> $MAILFILE
fi
else
WASDOWN=`grep "^$MACHINE:" $OLDLIST | wc -l`
if [ $WASDOWN -gt 0 ]; then
SENDMAIL=1
echo "$MACHINE now responding, down from `grep \"^$MACHINE:\" $OLDLIST`" >>
$MAILFILE
echo "`grep \"^$MACHINE:\" $OLDLIST` ---> $DATE" >> $LOGFILE
fi
fi
done


Sachin

[Santosh Nair_1]

This might be a bit overkill, but Big Brother does everything that you've asked and much more. You can get Big Brother from:

http://bb4.com

Also, since most of BB is shell scripts, its very platform independant. Hope this helps.

-Santosh

[Sridhar Bhaskarla]

Laurie,

You can follow this logic .

1. Your script is valid for the first time. But what you need to do is to take the last line of the output and put the time stamp with the server name in a history file. tail -1 does the trick here.

2. Again, run your script. Now you verify this last line with that is already there in the history file. If they match, then don't send page. Else send page, and keep the new entry in the history file. You can use this history file later to calculate the downtimes. Or you can delete the entry if you don't want.

3. Repeat step 2 for all the entries.

The script would be like this. You may need to modify it.

#!/usr/bin/ksh
HIST=/var/adm/history
DATE=`date "+%b %d"`
HOST=`uname -n`
#ADMIN_PAGER=9131234420@messaging.sprintpcs.com
ADMIN_EMAIL=lakrumrey@statestreetkc.com

# Check to see if any of the servers are going down (shutdown or reboot)

LINE=`grep -i "going down" /var/adm/syslog | tail -1`
MYHOST=`echo $LINE|awk '{print $4}'`
MYTIME=`echo $LINE|awk '{print $1$2$3}'`
grep $MYHOST:$MYTIME $HIST > /dev/null 2>&1
if [ $? != 0 ]
then
echo "$MYHOST went down at $MYDATE" |mailx -s
"$MYHOST went down" $ADMIN_EMAIL
echo "$MYHOST:$MYTIME" >> $HIST
fi

-Sri

[harry d brown jr]

You might also want to add a feature that will resend the page/email if you don't acknowledge it within a certain time period. That will increase your chances of receiving a page, because we all know that pagers aren't 100% reliable.

There is also software that will leave you a voice mail, and with some phone systems you can have it page you when you get voice mail. They usually will continue paging you until you acknowledge it.

live free or die

harry

[Sridhar Bhaskarla]

Laurie,

You can follow this logic .

1. Your script is valid for the first time. But what you need to do is to take the last line of the output and put the time stamp with the server name in a history file. tail -1 does the trick here.

2. Again, run your script. Now you verify this last line with that is already there in the history file. If they match, then don't send page. Else send page, and keep the new entry in the history file. You can use this history file later to calculate the downtimes. Or you can delete the entry if you don't want.

3. Repeat step 2 for all the entries.

The script would be like this. You may need to modify it.

#!/usr/bin/ksh
HIST=/var/adm/history
DATE=`date "+%b %d"`
HOST=`uname -n`
#ADMIN_PAGER=9131234420@messaging.sprintpcs.com
ADMIN_EMAIL=lakrumrey@statestreetkc.com

# Check to see if any of the servers are going down (shutdown or reboot)

LINE=`grep -i "going down" /var/adm/syslog | tail -1`

if [ $LINE ]
then
MYHOST=`echo $LINE|awk '{print $4}'`
MYTIME=`echo $LINE|awk '{print $1$2$3}'`
grep $MYHOST:$MYTIME $HIST > /dev/null 2>&1
if [ $? != 0 ]
then
echo "$MYHOST went down at $MYDATE" |mailx -s
"$MYHOST went down" $ADMIN_EMAIL
echo "$MYHOST:$MYTIME" >> $HIST
fi

fi
-Sri

[Craig Rants]

Laurie,

Try this page, you will be able to customize this software and it will provide you with alot of future functionality.

Enjoy,
Craig
http://www.cert.dfn.de/eng/logsurf/

[John Payne_2]

If ytou get tired of your scripting, you might also try looking at netsaint. It does more than just a ping sweep. You can get all kinds of info out of the system agents. Runs best on linux, though...

[Laurie A. Krumrey]

Hi Sachin,

I am also using your ping script. Everything works when my server is down. How do I get it
to just send me one email and not keep repeating itself?

What did I miss?

for MACHINE in $MACHINE;
do

RESULT=`$PING $MACHINE $BYTES $COUNT | grep "packet loss" | sed 's/%//g' | awk '
{print $7}'`

if [ "$RESULT" = "" ]; then
echo "Unexpected ping response from $MACHINE"

elif [ $RESULT -ge 100 ]; then
WASDOWN=`grep "^$MACHINE:" $OLDLIST | wc -l`

if [ $WASDOWN -gt 0 ]; then
grep "^$MACHINE:" $OLDLIST >> $NEWLIST

echo "Cannot ping $MACHINE" | mailx -s "$MACHINE NOT RESPONDING" $ADMIN_EMAIL
fi
else

WASDOWN=`grep "^$MACHINE:" $OLDLIST | wc -l`

if [ $WASDOWN -gt 0 ]; then
SENDMAIL=1

echo "$MACHINE now responding, down from `grep \"^$MACHINE:\" $OLDLIST`" |mailx -s "$MYHOST Down" $ADMIN_EMAIL

echo "`grep \"^$MACHINE:\" $OLDLIST` ---> $DATE" >> $LOGFILE
fi
fi
done

[Bernie Vande Griend]

Another idea I haven't seen mentioned yet, if you just want to send out a message when a server is being rebooted/shutdown, you could simply put a script in /sbin/rc3.d/K100goingdown. Then that script could send out your message. This wouldn't have to be in rc3.d, it could be in rc2.d as well, just make sure it starts with a K and is listed before the network stuff is shutdown.

One more option, if you have a central server that is consolidating syslog stuff, why not configuring syslog.conf on your server to send all its messages to that other server. Then you can set up a script on one server to look for errors/messages from all your servers. There are several good scripts out there search through a log file and filter out what you want.

That said, having some kind of real-time monitoring system as is mentioned above is still the best way to know when your server is down, as systems as known to crash or go down ungracefully without going through the proper shutdown sequence.

[Sachin Patel]

Hi Lauri,
You need simple trick.

#folloing .new and .old file will do that trick
NEWLIST="/tmp/server_ping.new"
OLDLIST="/tmp/server_ping.old"
MAILFILE="/tmp/server_ping.mail"
MAILUSERS="sachin"
SENDMAIL=0
BYTES=64
COUNT=4
DATE=`date`

PING=/etc/ping

DIRNAME=`dirname $0`
MACHINES="server1 server2 server3...."
LOGFILE="$DIRNAME/uptime.log"

# If there is no old list file create one
if [ ! -f $OLDLIST ]; then
echo "" > $OLDLIST
fi

echo "Current Time: $DATE" > $MAILFILE
echo "" >> $MAILFILE
echo "" > $NEWLIST



# Where mail is required
if [ $SENDMAIL -eq 1 ]; then
echo "" >> $MAILFILE
echo "Machines currently down:" >> $MAILFILE
cat $NEWLIST >> $MAILFILE
mailx -s "Server Uptime Change" $MAILUSERS < $MAILFILE
fi

#Update old list file
cp $NEWLIST $OLDLIST

Now you have $OLDLIST file and so it will not send you eamil again for that system. It will send you email when that system is up again or other system goes down.

Sachin