notification script

[Endrin]

Hi All.
Could someone help me to write a script on a hpux 11i. All it has to do is running one time per hour and giving the only the command "pmi -f" and if it doesn't return an output i can receive an printout like "application has stoped" and receive an email.
I'm not so familiar with scripting.
Thanks

[James R. Ferguson]

Hi Endrin:

Here's a general purpose process monitor. You can create a 'cron' task for it. Assume that you want to look for a process called "myproc" and mail/page if it isn't running:

# [ -z "`UNIX95= ps -C myproc -o pid= -o comm=`" ] && mailx -s "myproc" is not running!" root < /dev/null

Change the 'myproc' to match the basename of the process that you're interested in monitoring. As shown, this generates a mail to 'root'.

For example, since you want to watch a process called "pmi", substitute that string for "myproc" in the above code.

Regards!

...JRF...

[Endrin]

Hello James,
I just want to be more specific on this. The "pmi -f" lists a group of processes of the main application. When the application freeze up this command couldn't be executed. When I enter this command in the moment the system has hanged it stays on hold without giving a printout of the processes running. In that moment I can understand that the system has stopped and I want that the script gives the command "pmi" and if it returns a printout it says ok otherwise if the command doesn't return anything and stays on hold, the script wait for 30 sec and if he doesn't get a printout he send an email or a message and says that the system has stopped.
Thanks

[A. Clay Stephenson]

Here is a very handy-dandy Perl script for executing commands with a timeout. The command will exit with a status of 14 (SIGALRM) is the command times out. The signal handlers within Perl make it one of the best ways to do this from within a scripting language.

In your script that is invoked by cron you should set and export the PATH that contains
your "pmi" executable. You should also make sure that you have a symbolic link in /usr/bin that points to the "real" perl exectutable.

e.g. ln -s /opt/perl/bin/perl /usr/bin/perl

Your cronned script should look something like this:

#!/usr/bin/sh

PATH=${PATH}:/usr/bin:/xxx/yyy
export PATH

# NOTE: /xxx/yyy is the directory that holds
# the pmi command; you should also set and # export any env variables needed by the
# command

typeset -i STAT=0

timeoutcmd.pl -t 30 pmi -f > /dev/null 2>&1
STAT=${?}
if [[ ${STAT} -ne 0 ]]
then
echo "Pmi failed; status ${STAT}" | mail mickey@mouse.com
fi
exit ${STAT}

[Joseph C. Denman · Accepted solution]

Since "pmi -f" hangs, you may have to do something like:

!#/usr/bin/sh
#
#Set your Environment
pmi -f > /tmp/app_test.dat &

#If PMI Hangs, it should not write the file.
if [ -f /tmp/app_test.dat ]
then
rm /tmp/app_test.dat
else
mailx -s "App TEST Failed" admin@mydomain.com
fi

Something to start with anyway.

...jcd...

[Joseph C. Denman]

Put a

sleep 30 after

pmi -f > app_test.dat &


...jcd...

[James R. Ferguson]

Hi (again) Endin:

Here's a simple Perl script to alert you to your hung command:

# cat ./mympi
#!/usr/bin/perl
use strict;
use warnings;
my $cmd ="pmi -f";
my $tmout = $ARGV[0] =~ /\d+/ ? $ARGV[0] : 30;
eval {
local $SIG{ALRM} = sub { die "alarm call" };
alarm $tmout;
system("$cmd");
alarm 0;
};
alarm 0;
die "time-out!" if $@ && $@ =~ /alarm call/;
1;

...Run as:

# ./mypmi || mailx -s "Application stopped" root < /dev/null

...If your command doesn't finish in 30-seconds (by default) you will get an email alert notification. You can dynamically change the timeout by specifying the number of seconds at runtime:

# ./mypmi || mailx -s "Application stopped" root < /dev/null

Regards!

...JRF...

[Bill Hassell]

And just a sanity check on running a script like this. If you decide to check every 5 minutes, be prepared for hundreds of emails on a weekend or holiday unless you have staff available to read the email 24x7. It is common to run these types of scripts and forget that the script will keep sending email forever until the problem is resolved. This can be very expensive if the destination is a cellphone or text pager.

To avoid this type of problem, run the script with the response time under consideration. In other words, if it takes 30 minutes to locate someone to fix the problem, don't run the script more than once every 15-20 minutes. Be sure to consider problems during the weekend too.

[Endrin]

It works. Thank you all.
I will assign the respective points to each of you.