crond stopped

[cl cheung]

Hello, we have a system of version HPUX 10.20. The crond was found disappeared for about once every few months. We have not changed anything on the system and the cron table.
We have found nothing related in the syslog.

Can you suggest how to diagnose this problem ?

Or if any patches applicable ?

If it is not solvable, we plan to write another script to auto-restart crond if it goes away. Any advise ?

Best regards
CL

[Mark Grant]

I have never seen "cron" die of its own accord. I would start by seeing if someone is actually killing it by mistake.

Have a look at /var/adm/cron/log and see if you get any clue in there.

[cl cheung]

So far we have not able to collect anything useful from the cron log. This is a server system for special purpose application and usually no one is allowed to touch the system. The possibility of being killed by human is quite low.

I've read some other posts about the apmd in Linux that may kill the crond if the system is running in battery mode (and crond is configured to be killed when low power).

Can you relate anything like this in HPUX and I should pay attention to ?

If not, we plan to write a script to restart the crond automatically. Any comment on it and any sample script for this purpose to share with us ?

Best regards,
CL

[T G Manikandan]

With looking at the cronlog do check your cron scripts as whether any one of those scripts are the culprits.

check which cron script is run that time interval.

check any other messages in syslog.log like a syslogd stop.

Revert

[T G Manikandan]

check for the cron patch too

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.main|patch.breadcrumb.search|&patchid=PHCO_27422&context=hpux:800:10:20

[Mark Grant]

Linux does kill things off if you start running out of virtual memory, HPUX doesn't, it just tends to just get really slow and potentially become unusable. It is a different strategy, not worse, just different.

First thing I would suggest for your "cron" monitoring job is not to run it from cron :)

Secondly, I don't like the idea of monitoring cron using the "ps" there are already too many "ps" jobs running around.

In your case, it would be a good idea if your script kept a log of each time it restarted cron. That way, you might determine what it is that is killing cron and therefore not need to monitor it at all.

I would actually get cron to sort of monitor itself. Create a cron job to "touch" a file like "/tmp/cronalive" every five minutes(you could even "touch" your own shell script file).

Then have your shell script a) exit if it is already running and b) check the modification time of the file and then sleep for 5 minutes. If the file is over five minutes old then cron has probably died you can check at this point with "ps".

The reason to get the script to exit if it is already running is so that you can get cron to attempt to restart it every five minutes too. That way you get the script restarting cron if it dies and cron restarting the script if it dies.

[Elmar P. Kolkman]

Have you seen any pattern in the daemon dying? It could be a kill of a parent process in your cron scripts, for instance...

I don't know if anyone tried it, but you could try using init to restart cron if needed...

[cl cheung]

>With looking at the cronlog do check your cron scripts as whether any one of those scripts are the culprits. check which cron script is run that time interval.

Do you mean that a cron script can kill the cron daemon ? We have quite a number of script running but they do not kill any process purposely.

> check any other messages in syslog.log like a syslogd stop.

We do not find anything like the syslogd stop. In case we find it later, what does it sugguest ?

[cl cheung]

> Have you seen any pattern in the daemon dying? It could be a kill of a parent process in your cron scripts, for instance...

There is no special pattern. The recent instance was happening at the time that the system was not busy. Looks like only the cron daemon was killed. The cron tasks are normal application tasks and do not have any code to kill other process purposely.

> I don't know if anyone tried it, but you could try using init to restart cron if needed...

Can you elaborate how to use init to do this ? Is it safe and easy to do so ?

[Ralph Grothe]

Is your system under memory pressure?

Although Mark alleges that HP-UX never would kill crucial daemons such as cron, and in theory swap reservation should prevent this, I've experienced instances on few of our HP-UX boxes that occassionally did get under severe memory pressure where indeed daemons such as cron or sendmail disappeared.
At least what HP pretend, and how theory prescribes it, you may find hints on the OS's memory management from the documentation in /usr/share/doc/mem_mgmt*

[Elmar P. Kolkman · Accepted solution]

What you should run from init is a script, with respawn, that will run forever, checking for the cron process (look in /sbin/init.d/cron for the check done when stopping cron for the surest way to do this) and start it if it's not running anymore. And of course log that you've done it.

Because init will respawn it if the script stops, you're sure the monitoring always runs.

Mind that cron will get into background by itself when started.

Test your script and mind you don't have a path.

[Mark Grant]

Ralph,

I don't dispute that processes can die under heavy RAM pressure, I am saying that HPUX doesn't actively kill them off like Linux does.

Elmar,

Thanks for reminding me of the benefits of inittab. I often use this route myself but often forget it exists :) The only problem I can see with using inittab for cron though is that some things like to restart cron and they will fail if init is respawning them. Still could be the best option though.

[cl cheung]

In the last instance the cron was stopped in non-busy hour. Looks like the memory issue is not related.

So from the recommendations, I shall write a script, which installed in the inittab with respawn option, and monitor & restart cron from this script.

I will try this out if no further comments/findings from you experts.

Thanks a lot !

[Victor Fridyev]

Hi,

I saw crond faults on systems with /tmp 100% full. Could you check this on the suspicious system ?
Regards