Problems with package services

[Andrey T.]

Hello,

I have an issue with ServiceGuard configuration. There is an application that starts with the following command:

/etc/cmcluster/pkgmz/online /opt/java6 /opt/mz mzadmin platform

Now, this "online" script issues the following command and ends:

su mzadmin -c "export JAVA_HOME=/opt/java6; export MZ_HOME=/opt/mz;/opt/mz/bin/mzsh startup platform"

After that, if I check the process list what I can see that is running is this:

mzadmin 5442 5441 0 16:22:10 console 0:00 /bin//vmstat 5
mzadmin 5441 1 0 16:22:05 console 0:03 /opt/java6/jre/bin/IA64N/java -Djava.net.preferIPv4Stack=true -server -Xmx128M -Xms32M -classpath lib/picostart.jar:lib/ojdbc14

That JAVA process (5441) is the one that is running the application and the one I need to define as a service in SG so it may monitor its status.

I tried to define each of them as a service, but as the scrit after it runs the command in it, the node crashes.

Anyone has any idea on how to setup this kind of configuration so that SG may monitor and failover depending on that java process state?

I really appreciate any help that you may provide.

Thanks,

Andy

[Matti_Kurkela · Accepted solution]

Create a script that runs in an infinite loop, sleeping for a while and then checking for the presence of the java process. If the process is not there, the script must exit; if the process is OK, the script should go on to the next pass around the loop.

Set this script as the service, and start the java process using the package start-up script commands. The idea is to start the actual java process before the service that monitors it starts. The monitor script should exit only when something is wrong.

If you want the system to attempt to restart the java process for a limited number of times before failing over to another node, you must implement this in your service script: Serviceguard's restart logic for services is not applicable here, as it would attempt to restart the monitoring script, not the java process!

Your service script should probably also check for a service flag: e.g. the presence or absence of a file in some pre-arranged location. If the file exists, the script should bypass all checks and pretend that everything is OK. This would allow you to intentionally stop the java process without shutting down the entire package, making updates and other application maintenance actions much easier.

If you implement the service flag, you should probably also set up something that prevents you from forgetting to remove the service flag when your maintenance is completed.

It might be useful to make this reminder slightly obnoxious: for example, a cron job that sends mail to all sysadmins once an hour if the service flag is set, or even sends a "wall" message about the service flag to all the logged-in users in 10-minute intervals, if the service is really important.

MK