HP-UX serviceguard with oracle toolkit

[Mikey De]

Hi all

Need some help , anyone is using hp serviceguard with oracle toolkit

this is all new to us.

got a two node cluster , running service guard with three oracle databases with one listener

Out system administrator has set up a package for all the three databases and its is aware of the three databases by amending the sctipts.

what he tells me that if we want to do any maintenance on one of the databases we need to enable maintenance mode by creating a file called oracle.debug and this will stop the package failing over

But all monitoring will stop on the other databases until the oracle.debug is removed

Now this is what we do not want to happen, as if we are peforming maintenance on one databases we want service guard to still monitor other databases.


Anyone out there has any experience implementing service guard with oracle.

need your advise and help

Thanks
mike

[Matti_Kurkela]

Depending on the settings of the package, if the monitoring indicates a problem, Serviceguard might attempt to forcefully kill all the processes accessing the package filesystems and failover the package to another node. If this happens in the middle of your maintenance action, this might cause some damage to the database (depending on what you're doing with it). You probably don't want this.

You're now seeing why adding too many things into a single package is not a good idea: when one detects a problem, all will failover - always.

As you're doing maintenance on one of the database, that sort of implies there might be at least one DBA or equivalently-skilled person present while you're doing your maintenance. I suppose s/he might be relied on to fix any problems that might happen on the other two databases, while the third database is "down" for maintenance, or to use his/her judgement to decide whether the problem is so bad the maintenance must be aborted.

The real solution to your problem would be separating the databases to their own packages.

Serviceguard monitors things through the "services" defined in package configuration: if the service process (which might often actually be the monitoring script) stops for any reason, Serviceguard can optionally attempt to restart it first, or just proceed to package failover.

Find out what services are configured in your package and what they do. Once you understand those, you'll be in a better position to plan your next actions.

MK

[Mikey De]

MK

Thank for your reply, If i repeat my self i do apologise.

What i have read , correct me if i am wrong is that if you are in maintenance mode serviceguard stops
monitoring all the databases and will not failover until you come out of maintenace mode.

As you said we dont want the above setup as if there is a problem with one of the other two databases while we are doing maintenance
on the thrid databases it will not restart failover and there will be no way of knowing unless the users complain.


Also I have been told that creating seperate packages for each database you need seperate ip addresses etc that why our
unix admin did not have seperate package for each database. to have seperate packages do we need anything different?


Ideally what we want is that if we are doing maintenance on one of the databases we still want SG to monitor the other two databases and if there
if there is a problem restart the database on the same node and if that is not possible failover to the passive node.

So any experience and ideas would be appreciated.

Thanks
Mike

[Matti_Kurkela · Accepted solution]

> What i have read , correct me if i am wrong is that if you are in maintenance mode serviceguard stops monitoring all the databases and will not failover until you come out of maintenace mode.

Such a "maintenance mode" is not a fundamental feature of Serviceguard, but it's easy to implement in monitoring scripts. By your description, I think I can roughly guess what your sysadmin has done (I think I read about the technique here in ITRC Forums, and have since then implemented it a few times), but beyond that, I only know what you've told me.

When the specified file exists, the monitoring script skips all the tests it would normally make and behaves as if everything is OK.

But if the entire node would crash or lose network connectivity, then Serviceguard would still failover the package, unless Serviceguard cmmodpkg command was used to explicitly disable package failover.

> Ideally what we want is that if we are doing maintenance on one of the databases we still want SG to monitor the other two databases and if there is a problem restart the database on the same node and if that is not possible failover to the passive node.

There are two problems with that:

1.)
According to documentation, the standard Serviceguard Oracle toolkit won't ever try to restart the database on the same node, probably because restarting Oracle can take a long time, and Oracle generally doesn't just crash without a good cause.

(See: http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02041313/c02041313.pdf page 8)

If your sysadmin has added some extra functionality, I cannot say anything about that without at least seeing the scripts.

2.)
Since all three databases are in one package, it is not possible to failover them one by one: either all three databases will failover, or none will.

This is a fundamental design principle of Serviceguard: a package is the smallest unit that can failover independently.


> ... if there is a problem with one of the other two databases while we are doing maintenance on the thrid databases it will not restart failover and there will be no way of knowing unless the users complain.

"No way of knowing"???

A good sysadmin or DBA should be able to write a quick script for basic database aliveness test in less than fifteen minutes or so.

Something like this, perhaps:


#!/bin/sh
export ORACLE_SID=
while true
do
clear
sqlplus -s /nolog <whenver sqlerror exit failure rollback;
connect ;
select '1' from dual;
EOF
sleep 15 # time in seconds between tests
done

(the parts depend on your database configuration)

Once started, the script will repeat forever until interrupted. In 15-second intervals, it will connect to the database and perform a very simple operation that should always be successful. If it starts producing error messages instead, something is wrong with the database.

Your DBA probably already has some tool that can do the equivalent of this and more.

> Also I have been told that creating seperate packages for each database you need seperate ip addresses etc that why our
unix admin did not have seperate package for each database.

Yes, that is correct. Transitioning from your current configuration to one-database-per-package configuration is going to be a major operation, requiring careful planning and some application configuration changes (to make them connect to new database packages correctly).

Sorry, but I don't see any other way to fulfill your requirement that a maintenance to one database must not have any effect on the monitoring and failover of other databases. Your environment has a design problem; it is better to take the trouble of fixing that once and for all, than to keep trying to find workarounds to all the secondary problems it causes.

MK

[Mikey De]

Thanks MK

help is appreciated. it give me something to think about

our unix administrator has managed to amend the toolkit scripts so that if the maintenance flag is set for one database SG still monitors the other databases.

Mike

[Mikey De]

info provided from MK has pointed me in the right direction

[Terry.giblin]

Two nodes, three Databases, one package, limited options.

Three nodes, three databases', multiple packages, multiple failovers, are much more fun.

How would you do it then, I wounder?

Where is the manual?

Terry Giblin