Capturing hung command in a script.

[sbahra]

We are working on a script which needs to monitor if APPHA is hung or not. For this, we manually have to run command "hastatus -sum" If there is an output out of this command then we assume that the APPHA is still running and if the command hangs then it means that APPHA is hung.

What string can be added in a the script to check if "hastatus -sum" command has hung or not? If hung then it should send out an email.

[Matti_Kurkela · Accepted solution]

If you run "hastatus -sum" normally in your script and it hangs, the script won't proceed either. You'll need to implement a timeout  that stops the hanging command and causes the mail to be sent if the command does not complete in a reasonable amount of time.

If you Google with keywords "shell script timeout", I'm sure you will find many examples.

If "hastatus -sum" provides a non-zero return status (also known as return code or exit code) if it's interrupted and zero if it completes successfully, something like this might do the job:

#!/bin/sh

WAITTIME=20 # seconds to wait for hastatus -sum to complete

# run the hastatus command in the background, discarding any output
hastatus -sum >/dev/null 2>&1 &
HASTATUS_PID=$!

# set up a timeout that will kill the hastatus command when 
# $WAITTIME seconds has passed, unless it has completed before that.
(sleep $WAITTIME; kill $HASTATUS_PID 2>/dev/null) &
TIMEOUT_PID=$!

# wait for the hastatus command to either complete or get killed; read its return status
wait $HASTATUS_PID
RESULT=$?

# if the timeout is still running, stop it (ignore any errors)
kill $TIMEOUT_PID 2>/dev/null

# read the return status of the timeout process (we don't need it 
# but running the wait function prevents it from remaining as a 
# zombie process)
wait $TIMEOUT_PID

if [ $RESULT -ne 0 ]
then
    echo "something is wrong with APPHA, result was $RESULT"
    # insert the command to send mail here
else
    echo "APPHA seems to be OK"
fi

[sbahra]

Just 1 word... Brilliant!!! Thanks a lot Matti.. it works like a wonder for me.