How to handle dependencies between packages

[Christian Schulze]

Hi folks,

I have a problem, with our Serviceguard-Clusters.
There is on Oracle-Package configured which runs fine. Then there are several other packages with several applications each which always need to have access to a running Oracle (otherwise they crash).
So now I have configured the App-Packages to wait for oracle beeing available during Cluster-startup.

In order to shut down the Apps if the Oracle-Pack goes down, I have configured some cmhaltpkgs in oracle-controlscript.

This works perfectly, if the package/node/cluster is halted via the cm-commands.

Even if the node crashes where oracle and the packages are running I am fine.

But, If the Apps are on another Node than Oracle, there is the problem that the shutdown-control-script is not run -> Package stays up -> Most of the Apps crash. (and do not get restarted)

Now I am thinking about some Ideas to resolve this problem.

1a. rewrite all apps so they can deal with a temporarily unavailable DB (Well, not really possible)

1. configure all apps as serviceguard-monitored services (-> puh, lots of work, they change often), right now SG only runs a script, provided by the apps-Developer, where all apps of this package get started

2. setting up a monitoring package, which constantly monitors oracle, in case of failure kill and restart all the apps packages (this would allow to do some maintenance just by halting the monitoring pack)

3. Using some builtin methods of MC-SG which I dont know ?...
maybe there exist a package_fail_fast like node_fail_fast ? but could not find it in the docs.


T make things worse, we have another cluster, running packages which also rely on the Oracle-Package of Cluster One.


Is somebody out there having a similar problem (and maybe solved this already?)

Any tips are appreciated.

Christian

[Marvin Strong]

I think your stuck with one of your monitoring solutions. Last time I dealt with package dependancies, it was addressed with monitors.

[Steven E. Protter]

Shalom Chris,

Several responses.

If package a can not run without package b then I do the best I can to build package a and b into package c which includes both.

You can run any startup script you wish, including a custom one you wrote to bring up these two packages at the same time.

A good monitor script on a and b can make things reliable enough to avoid that.

There are probably other and better solutions than mine, but its worked for me from time to time.

SEP

[Prasanth B]

Hi,

I have a simple solution which I have implemented. In your oracle package control script, before starting oracle , check the status of other packages, if it is running, halt the dependeant packages, start up oracle, then startup the app packages.

This way if database server is recovering from a crash, it will restart the dependant applications.

[Christian Schulze]

Hi everybody,
thanks for the responses.

I did some testing, and it seem that I will try a solution like this.

Each package will du some monitoring of services it needs. (The actual monitoring is not yet done, maybe something like doing tnspings for example). if this fails, the package will execute:
nohup cmhaltpkg &
cmmodpkg -e

this allowed the package to initiate its move to the next available host.

This would even work for packages running on another cluster, and each package can decide what to do for itself. Oracle does not have to kill and restart other apps, does not have to check wether the package was down for maintenance and so on.

I will provide my solution here when I am done.

But Ideas and comments are still welcome ...


Christian

[Prasanth B]

Hi,

You got to be careful with the soultion that you just mentioned becuase it may cuase the packages to move around unnecessarily. As an example, assume that there was a network glitch of some sort which pervented the communication between apps and db, the packages will go crazy and start moving around.

-PB