cmrunserv under Service Guard

[David Macchion]

Hi,

Inside a package run script, I start up an application thru "cmrunserv -R my_app /my_app_full_path". This application cannot boot up - because of an incorrect SHLIB_PATH for example. The package is stated running as the run script goes thru. And although MC/SG keeps restarting it, the package status stays running. Normal feature ?

Thanks

[Brian Crabtree]

Check your "SERVICE_RESTART" in the package control script. If the setting is "-R" then it will continue to restart over and over.

Thanks,

Brian

[melvyn burnard]

You normally do not start an application using cmrunserv, as this is used to start services that would monitor the health of your application.
I would revisit how you activate your application and also ensure it has the correct SHLIB_PATH environment variable passed to it upon start up.

[Bernhard Mueller]

cmrunserv -R
means "restart service an infinite number of times *without* halting the package"
maybe you want to use
cmrunserv -r 3 to stop that after the third attempt and initiate failover....

interestingly, and in addition to what Melvyn said, the current man pages *no longer* include the sentence "cmrunserv is not intended for use in the shell ..."

Regards
Bernhard

[Stephen Doud]

Complex applications are normally launched by way of the customer_defined_run_cmds function in the package control script - NOT AS A SERVICE.

Generally, SERVICEs are non-terminating scripts or commands which are most often used to monitor the health of an application which is started via the "customer_defined_run_cmds" function in the script.

A package SERVICE is comprised of 3 components:
SERVICE_NAME
SERVICE_CMD
SERVICE_RESTARTS
cmrunserv uses all 3 customer_defined parameters to implement the service. ServiceGuard will monitor the PID of the service, restarting or terminating/failing-over the package if the service command terminates.

I have never seen an admin add another cmrunnserv command in a package control script to launch an application. As previously stated, use the customer_defined_run_cmds to start the application, and use the SERVICE_* parameters to monitor the application health by way of a monitor script.

-StephenD

[David Macchion]

Thanks!

OK: I have to revisit my design again then.

Basically, you're saying that a package has one and only one service and that service only monitors that the actual application processes are up. These processes are to be started in customer_defined_run_cmds.

But suppose I want to use the -r or -R feature. The monitoring script is then restarted by SG if I decide to exit when one process of the application goes down.
But I guess customer_defined_run_cmds is not rerun. So, don't I have both to start and then monitor in the monitoring script itself ?

More generally, this is the behavior I'm trying to implement:
- try to start the application. If it doesn't start right away (only one attempt), then fail the package without even trying on the alternate node.
- once the application is up and running, keep it up on that machine unless the machine goes down.

It's OK if I need to manually reset the RESTART counter sometimes. But I hope I won't have to code that logic in my monitoring script.

Thanks again,

David

[melvyn burnard]

>Basically, you're saying that a package has one and only one service and that service only monitors that the actual application processes are up. These processes are to be started in customer_defined_run_cmds.>
Almost correct, you can have multiple services to monitor your application's vital processes.

>But suppose I want to use the -r or -R feature.>
Yes you can.
>The monitoring script is then restarted by SG if I decide to exit when one process of the application goes down.>
Correct, if you find a process you are monitoring dies, you can attempt ro restart the p[roces and monitor again.
>But I guess customer_defined_run_cmds is not rerun.>
Correct
> So, don't I have both to start and then monitor in the monitoring script itself ?>
Not sure what you mean here, but you can call a separate monitor script from in the package control file, or put some logic in the control script for a monitor section, as is done in the newest Oracle Toolkit script, for example.

>More generally, this is the behavior I'm trying to implement:
- try to start the application. If it doesn't start right away (only one attempt), then fail the package without even trying on the alternate node.
- once the application is up and running, keep it up on that machine unless the machine goes down.>

Well that is something completely out of the ordinary, as normally you expect to be able to have the package switch if it fails on startup, as this could be due to many reasons. I believe you will have to writ eand play with your own loic to get that to work.

[David Macchion]

Thanks Melvyn!

1/
What I mean by "both start and then monitor in the monitoring script itself":

My application is started by customer_defined_run_cmds. Then it's monitored by my monitoring service. Ok, now one of its processes goes down. The monitoring script detects it. And exits... As I use a -r or -R, the monitoring script gets restarted by SG. But what restarts the application ?

Thus my question: do I need to restart the application in the monitoring script ? Just before I exit ? The logic would then be:

while 1 do
if "one process missing" then
start application
exit 1
fi
...
done

Well, maybe I can drop the customer_defined_run_cmds then. And have everything in the monitoring script. Like:

start application
while 1 do
if "one process missing" then
exit 1
done

I don't control exactly how much time I need to bring the application back up though. So multiple starts may step over one another...

2/
Why that behavior:

Well, because the environment around the application keeps changing, we are not that sure that is going to start up well... We don't want to wait for 3 - for example - attemps before SG states it can't run on that node.

On the other hand, once it's up and running, we have processes that can die on us often. It's usually fine to restart them invidually on the same box without disturbing the whole behavior of the app. Moving the app on an alternate node may cause a longer overall downtime - to other client apps. I'd say we need SG for the machine fail over feature only...

Maybe I'll end up coding something like:

start application
if one process missing
exit 1

count=0
while 1
if "one process missing" then
if count > 2 exit 1
else
count++
start application
fi
fi
done

Ugly ? Of course I'd drop -r or -R.

Yes, I'll have a closer look at the Oracle SG scripts.

Thanks everyone again.