topic Re: crond stopped in Operating System - HP-UX

crond stopped

cl cheung — Thu, 01 Apr 2004 01:28:30 GMT

Hello, we have a system of version HPUX 10.20. The crond was found disappeared for about once every few months. We have not changed anything on the system and the cron table.
We have found nothing related in the syslog.

Can you suggest how to diagnose this problem ?

Or if any patches applicable ?

If it is not solvable, we plan to write another script to auto-restart crond if it goes away. Any advise ?

Best regards
CL

Re: crond stopped

Mark Grant — Thu, 01 Apr 2004 01:33:19 GMT

I have never seen "cron" die of its own accord. I would start by seeing if someone is actually killing it by mistake.

Have a look at /var/adm/cron/log and see if you get any clue in there.

Re: crond stopped

cl cheung — Thu, 01 Apr 2004 01:43:36 GMT

So far we have not able to collect anything useful from the cron log. This is a server system for special purpose application and usually no one is allowed to touch the system. The possibility of being killed by human is quite low.

I've read some other posts about the apmd in Linux that may kill the crond if the system is running in battery mode (and crond is configured to be killed when low power).

Can you relate anything like this in HPUX and I should pay attention to ?

If not, we plan to write a script to restart the crond automatically. Any comment on it and any sample script for this purpose to share with us ?

Best regards,
CL

Re: crond stopped

T G Manikandan — Thu, 01 Apr 2004 01:52:14 GMT

With looking at the cronlog do check your cron scripts as whether any one of those scripts are the culprits.

check which cron script is run that time interval.

check any other messages in syslog.log like a syslogd stop.

Revert

Re: crond stopped

T G Manikandan — Thu, 01 Apr 2004 01:55:08 GMT

check for the cron patch too

http://www2.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.main|patch.breadcrumb.search|&patchid=PHCO_27422&context=hpux:800:10:20

Re: crond stopped

Mark Grant — Thu, 01 Apr 2004 02:03:52 GMT

Linux does kill things off if you start running out of virtual memory, HPUX doesn't, it just tends to just get really slow and potentially become unusable. It is a different strategy, not worse, just different.

First thing I would suggest for your "cron" monitoring job is not to run it from cron :)

Secondly, I don't like the idea of monitoring cron using the "ps" there are already too many "ps" jobs running around.

In your case, it would be a good idea if your script kept a log of each time it restarted cron. That way, you might determine what it is that is killing cron and therefore not need to monitor it at all.

I would actually get cron to sort of monitor itself. Create a cron job to "touch" a file like "/tmp/cronalive" every five minutes(you could even "touch" your own shell script file).

Then have your shell script a) exit if it is already running and b) check the modification time of the file and then sleep for 5 minutes. If the file is over five minutes old then cron has probably died you can check at this point with "ps".

The reason to get the script to exit if it is already running is so that you can get cron to attempt to restart it every five minutes too. That way you get the script restarting cron if it dies and cron restarting the script if it dies.

Re: crond stopped

Elmar P. Kolkman — Thu, 01 Apr 2004 02:55:49 GMT

Have you seen any pattern in the daemon dying? It could be a kill of a parent process in your cron scripts, for instance...

I don't know if anyone tried it, but you could try using init to restart cron if needed...

Re: crond stopped

cl cheung — Thu, 01 Apr 2004 03:03:59 GMT

>With looking at the cronlog do check your cron scripts as whether any one of those scripts are the culprits. check which cron script is run that time interval.

Do you mean that a cron script can kill the cron daemon ? We have quite a number of script running but they do not kill any process purposely.

> check any other messages in syslog.log like a syslogd stop.

We do not find anything like the syslogd stop. In case we find it later, what does it sugguest ?

Re: crond stopped

cl cheung — Thu, 01 Apr 2004 03:09:42 GMT

> Have you seen any pattern in the daemon dying? It could be a kill of a parent process in your cron scripts, for instance...

There is no special pattern. The recent instance was happening at the time that the system was not busy. Looks like only the cron daemon was killed. The cron tasks are normal application tasks and do not have any code to kill other process purposely.

> I don't know if anyone tried it, but you could try using init to restart cron if needed...

Can you elaborate how to use init to do this ? Is it safe and easy to do so ?

Re: crond stopped

Ralph Grothe — Thu, 01 Apr 2004 03:21:12 GMT

Is your system under memory pressure?

Although Mark alleges that HP-UX never would kill crucial daemons such as cron, and in theory swap reservation should prevent this, I've experienced instances on few of our HP-UX boxes that occassionally did get under severe memory pressure where indeed daemons such as cron or sendmail disappeared.
At least what HP pretend, and how theory prescribes it, you may find hints on the OS's memory management from the documentation in /usr/share/doc/mem_mgmt*

Re: crond stopped

Elmar P. Kolkman — Thu, 01 Apr 2004 03:28:22 GMT

What you should run from init is a script, with respawn, that will run forever, checking for the cron process (look in /sbin/init.d/cron for the check done when stopping cron for the surest way to do this) and start it if it's not running anymore. And of course log that you've done it.

Because init will respawn it if the script stops, you're sure the monitoring always runs.

Mind that cron will get into background by itself when started.

Test your script and mind you don't have a path.

Re: crond stopped

Mark Grant — Thu, 01 Apr 2004 04:02:51 GMT

Ralph,

I don't dispute that processes can die under heavy RAM pressure, I am saying that HPUX doesn't actively kill them off like Linux does.

Elmar,

Thanks for reminding me of the benefits of inittab. I often use this route myself but often forget it exists :) The only problem I can see with using inittab for cron though is that some things like to restart cron and they will fail if init is respawning them. Still could be the best option though.

Re: crond stopped

cl cheung — Thu, 01 Apr 2004 04:29:08 GMT

In the last instance the cron was stopped in non-busy hour. Looks like the memory issue is not related.

So from the recommendations, I shall write a script, which installed in the inittab with respawn option, and monitor & restart cron from this script.

I will try this out if no further comments/findings from you experts.

Thanks a lot !

Re: crond stopped

Victor Fridyev — Thu, 01 Apr 2004 05:05:37 GMT

Hi,

I saw crond faults on systems with /tmp 100% full. Could you check this on the suspicious system ?
Regards