topic Re: catching process death in Operating System - HP-UX

catching process death

Ignacio Javier — Fri, 30 Mar 2007 06:25:37 GMT

Hello gurus:

I would like to ask you if you konw a development technique or if the hpux11.23 SO has some way to cath a process, not controlled death.

What i want is to develop a process than knows inmediatly when some others has dead.

Thanks in advance

Re: catching process death

SANTOSH S. MHASKAR — Fri, 30 Mar 2007 06:45:01 GMT

Hi,

Pl. elaborate what exactly u want to do.

Re: catching process death

Ignacio Javier — Fri, 30 Mar 2007 06:51:38 GMT

Ok, what i want is to develop a process to control others, to start them if they die.

IÂ´ve got now one that does it sending a signal every 5 seconds and i want to improve it, so they intercomunicate or some other way within SO, so the controlling process knows when the other die, just in the moment it takes place.

Regards

Re: catching process death

Veeru_1 — Fri, 30 Mar 2007 07:59:19 GMT

Hi Ignacio,

I am developing a new tool - sentinel - which does exactly that. It is available for beta users with WDB 5.6 available for free download from the web at http://www.hp.com/go/wdb. The tool is not yet supported by HP since we first want to validate its need.

Please take a look at it. The specific usage that you want to try is

$> /opt/sentinel/bin/sentinel -silent -exit mail

You'd have to specifiy SENTINEL_MAIL env variable to the appropriate email address (it defaults to root@localhost).

Instead of using the 'mail' action for the 'exit' event, you can also specify some other command using the '-cmd ' action. For example, if you want a script test.sh to be executed automatically when the process dies, do

$> /opt/sentinel/bin/sentinel -silent -exit -cmd test.sh

Sentinel either spawns a process if you specify the command with arguments, or it takes a pid.

Other events of interest to you would be 'errexit' (non-zero exit code) or 'abort' (uncaught signal).

For more events and actions, do

$> /opt/sentinel/bin/sentinel -help

or look at the man page after adding /opt/sentinel/docs/man/man1 to MANPATH env variable.

Please inform me if it works ok for your case by sending an email to Veeru at hp dot com.

Cheers!
Veeru

Re: catching process death

Ralph Grothe — Fri, 30 Mar 2007 08:25:21 GMT

I think to have read about Sentinel, which Veeru recommends, in some Linux context not so long ago.
Sounds promising.
But there are numerous ways of monitoring processes.
For your situation probably overkill, but I for instance have many vital processes monitored by my Nagios server.
There comes a handy Nagios plug-in called check_procs with quite a few options.
If you define a Nagios event_handler and write a wee wrapper script you can even have crashed procs be restarted through the event handler by Nagios unattendedly, but getting notice of it however.
Albeit, one should always determine the cause for a crashed process rather than having it simply restarted automatically.

Re: catching process death

A. Clay Stephenson — Fri, 30 Mar 2007 08:59:32 GMT

One word: SIGCHLD.

Re: catching process death

Ralph Grothe — Fri, 30 Mar 2007 09:12:22 GMT

That's fine if the monitoring process is the parent process, so that it actually can catch a SIGCHLD to get the child's exit value through waitpid.
But what if the monitor or agent is a completely unrelated process?

Re: catching process death

Patrick Wallek — Fri, 30 Mar 2007 09:26:01 GMT

If you have several unrelated processes that you are trying to keep running, why not start them via /etc/inittab and use the respawn option. That way if they die, they get restarted automatically.

Re: catching process death

Emil Velez — Fri, 30 Mar 2007 20:17:51 GMT

depending on how you have measureware configured it will capture how much resources a proces has used in that minute. Measureware collects data on interesting processes and can collect data on processes when they start, when it dies and if a process exceeds a threshold. This is configured in a file called /var/opt/perf/parm assuming that you have measureware (openview performance agent) installed and configured.

You can have a program do the extract command to watch for the data on that process.

Another option would be to configure glance with the advisor mode to print out the process list and look for a state of died for that process name.

Re: catching process death

A. Clay Stephenson — Fri, 30 Mar 2007 20:42:50 GMT

Since this question was asked in the context of a development technique then it naturally fits into the concept of a parent process which fork()'s and exec()'s a child proces to do the actual work while the parent just sits there (okay wait()'s) with a SIGCHLD handler to notify it when the child terminates or receives a signal. I see no reason to reinvent the wheel for something that UNIX has been able to do since it was a wee babe.

Re: catching process death

Ralph Grothe — Mon, 02 Apr 2007 05:31:37 GMT

It hasn't been made quite clear if the controlling process really is the parent.
To me it sounded rather unrelated.

"Ok, what i want is to develop a process to control others, to start them if they die."

Sure, one could as Patrick suggested regard init as parent of all, and thus let the moitored processes be respawned by init.
But for a multitude of processes or services this hack doesn't sound too flexible to me.

Coincidentally, I discovered in an old issue of the Linux Journal that I bought last weekend from my local tech book shop for a tenth of the issuing price, a related article that suggests a combination of a tool called "smart" together with various Nagios plug-ins in case of more involved service checks.
In that article the following link was given:

http://www.linuxjournal.com/article/9268

I haven't yet played with Smart but it sounds more lightweight than a full fledged Nagios installation.

Re: catching process death

Veeru_1 — Mon, 09 Apr 2007 13:19:37 GMT

Were you able to, or do you plan to use Sentinel? Please tell me if so. I'm eager to find out about your experience in using it.

Thanx!
Veeru