how to find cron freeze/hung status?

[Deva_1]

Hi,
I have few jobs are running under cron in HPUx box. how can i find If cron job hungs/freeze?
I have Cron process monitoring.., i am still wondering some times cron process shows in ps -ef |grep
but it wont run any sheduled cron jobs..,

does anyone has any logic to share with me?

Regards
Deva

[Michael Tully]

The first place to look is in the cron log. This will tell if the job has been submitted. If you find "rc=??" it may not have run at all. Normally cron does not hang, but may terminate, so there would be no process at all.

Log is in /var/adm/cron/log

[Dave Olker]

Hi Deva,

Another idea would be to have the cron jobs log their activity and output to a log file, or series of log files, and then you should be able to check the log files each morning to see what executed.

I recommend at least adding some print statements to the beginning and end of the cron jobs, but if you have the time and energy, you can add more print jobs at critical points during the cron job so that the log file will show the activity of the job as it progresses.

Depending upon how well you instrument (i.e. add print statements) your cron jobs, this logging information will also help you pin point exactly where a problem in the job occurred; or - in the case of a job hang - at what point the job hung.

Regards,

Dave

[Jeroen Peereboom]

Deva,

indeed you can use cron's logfile. It's a bit hard to read though. It's not difficult to find jobs that ended abnormally ('rc= ' lines) but it may be difficult for you to find which jobs have not ended at all.

The problem is within the jobs itself. They should be well-written. For example they should use exit codes.

I also remember someone wrote a C-program that will start a job and kill it or send an alert if the job does not finish within a certain period.

I always found the cron daemon to be reliable. Do you really have cron jobs that don't get scheduled by the cron daemon? Can you give us examples? Will the job never be run? Can you give us the 'crontab -l' output and a part of the cron logfile?

JP.

[Deva_1]

Hi All,
Thanks for the Inputs,

I would like to develop a script to monitor the cron process state (i.e. when it is in hung state) and also when is not updating cron logfiles, which means it is not executing any of the schedule jobs.

Sometimes, cron process will be running but it never runs scheduled cron jobs and also never updates cron logfiles

Thanks & Regards

Dev

[John Palmer]

One thing to be aware of with cron is that it imposes limits on the number of jobs that it has running concurrently. Check the man pages for details.

If you use cron to start jobs which run for a long time you can get into the state where cron won't start a new job until one of the current ones has finished. You can get round this by increasing the concurrent limit (see man pages) or possibly by changing the way your long running jobs work. Get the script that cron runs to kick off the real batch job and then exit for instance.

You can check what current cron jobs are active with the ps command. First find out the PID of the cron process ps -ef|grep cron. Then run ps to find processes whose PPID (parent process id) is that of cron.

Hope this helps,
John

[Simon Hargrave]

To monitor whether cron is working: -

Have a script that runs via cron every 15 minutes that does: -

#!/usr/bin/sh
if [ ! -f /tmp/cronmonitor.log.last ]; then
touch /tmp/cronmonitor.log.last
fi
date >> /tmp/cronmonitor.log


schedule it in cron with: -

0,15,30,45 * * * /usr/local/bin/cronmonitor.sh


Then have another script running permanantly in the background that does: -

#!/usr/bin/sh
while [ true ]; do
diff /tmp/cronmonitor.log /tmp/cronmonitor.log.last 2>&1 >/dev/null
if [ $? -eq 0 ]; then
echo "cron has stopped working" | mailx -s "CRON Failure" you@youraddress.com
fi
cp /tmp/cronmonitor.log /tmp/cronmonitor.log.last
sleep 1800
done


What this will do is change the cronmonitor.log every 15 minutes via cron. The other script will check every 30 minutes with diff that there has been a change. if there hasn't, it'll send an email. when it's checked it copies cronmonitor.log to cronmonitor.log.last to reference in the diff for the next check.

Simple but effective.

[John Palmer]

Personally, I've always found cron to be very reliable. On many occasions I've had to have checker scripts for ddodgy applications but never for cron itself.

[Simon Hargrave]

Why not have a cron job to periodically restart the cron daemon :) :) :)

Seriously though, as John says I've not really had reliability problems with the cron daemon itself.

[Geoff Wild]

Set all your jobs in cron to output to a log file first:

Most people do this:

# Run the sysinfo script
0 6 1 * * /usr/local/sysinfo/sysinfo -a -b -o /usr/tmp/`hostname`.sysinfo >/dev/null 2>&1

Instead do this:

# Run the sysinfo script
0 6 1 * * /usr/local/sysinfo/sysinfo -a -b -o /usr/tmp/`hostname`.sysinfo >/tmp/sysinfo.cron 2>&1

Then, after cron hangs, check all the /tmp/*.cron files for errors...

The other thing to check - patches - what OS are you at and are you uptodate with patches?

Rgds...Geoff

[Jeroen Peereboom]

Deva,

As other people pointed out you can write some programs to monitor cron's behavior.

I would suggest you only write these programs to find the root cause of your problem, such that in the end you don't need those additional programs any more. Under normal conditions cron should just work without any problems...

JP.

[Bill Hassell]

cron simply runs. It schedules the job but does NOT run it. That is a job for the shell, interpreter or the program. How do you know that cron did not run a job? Any job that has errors may abort immediately and cron just keeps running. Start with root's email: are there a bunch of cron error messages? Look at cron's log to see that each job was actually started. Most cron problems are due to wrong environments. Your script runs fine when you are logged in but cron never logs in and has a VERY small environment. The lack of customary values usually causes scripts to fail because they assume a full environment. If you really want to verify that a script has run, use logger at the beginning of the script, something like this:

logger -p local5.info -t cronchecker "$0 has been started"