Service Guard Oracle 8.1.6 Using standby

[Todd Legere]

Implementing Oracle standby server. Two N4000 running HP/UX 11.0 64 bit. Node A is primary. Node B is running standby.

Looking for suggestions for monitoring incase just standby node halts, and any helpful hint or templates the community can share on this one.

[James R. Ferguson]

Hi Todd:

One simple modification you can make is to create an email/pager notification to yourself whenever a failover occurs.

In the package control script in the 'customer_defined_run_cmds' function, add your email/pager notice. Do the same in the 'customer_defined_halt_cmds' function as you see fit.

...JRF...

[Todd Legere]

Have HP Openview and plan on adding monitor on the standby node. The pickle of the matter is that if the standby node goes down, there is a lot of manual intervention to resync just the standby node.

[Chris Stamps_1]

You might want to kick a cron on the primary to monitor the status through the cmviewcl command. Maybe execute a remsh to verify that the oracle standby server is up as well, and have that emailed to you. Or instead of cron, configure it as a service in the package.

[Carsten Krege]

You can monitor cluster resources (status of the local node, packages, cluster) through EMS' (Event Monitoring Service) clustermond monitor.

This (free) software allows you to send you a notification in case of a monitored event to syslog and textlog and as email, tcp message, snmp trap or console message.

For documentation see, Chapter 6 "Monitoring Cluster Resources" of
http://docs.hp.com/hpux/pdf/B7612-90015.pdf

(It is good to have latest EMS and SG patches installed.)

But in fact using cmviewcl for evaluation of node and package status is pretty sufficient, as long as you don't run it too often (not more than once a minute).

Carsten

[Mark van Hassel]

Hi Todd,

Whenever the standby host fails messages will show up in the syslog.log of all other cluster nodes, in this case your primary server:

syslog.log:Jul 30 17:26:38 primaryhost cmcld: Timed out node standbyhost. It may have failed.
syslog.log:Jul 30 17:26:38 primaryhost cmcld: Attempting to form a new cluster
syslog.log:Jul 30 17:26:53 primaryhost cmcld: Obtaining First Dual Cluster Lock
syslog.log:Jul 30 17:26:54 primaryhost cmcld: Obtaining Second Dual Cluster Lock
syslog.log:Jul 30 17:26:55 primaryhost cmcld: Turning off safety time protection since the cluster
syslog.log:Jul 30 17:26:55 primaryhost cmcld: may now consist of a single node. If ServiceGuard
syslog.log:Jul 30 17:26:55 primaryhost cmcld: fails, this node will not automatically halt
syslog.log:Jul 30 17:28:47 primaryhost cmcld: 1 nodes have formed a new cluster, sequence #4
syslog.log:Jul 30 17:28:47 primaryhost cmcld: The new active cluster membership is: primaryhost(id=1)

Just define these messages in your monitoring tool.

[melvyn burnard]

You could also take a look at the ServiceGuard Manager product available on http://www.software.hp.com
It is free! The manuals are available at docs.hp.com
Currently it is only useable as a monitoring tool, but will eventually be erquipped to move pkgs, halt nodes etc.
The client sw can run on Windows NT/98, HP-UX, and as of the new version A.01.02 on Redhat Linux

[Andreas D. Skjervold]

Hi

This might be a little off the track but just a thought anyway.

I use an Oracle Management Server with an Enterprise Manager Console hooked up to it.

With an Intelligent Agent running on the node, you can monitor virtually anything you want (db down, block corruption, space management issues, performance etc).

Monitoring is set up as Events which trigger e-mail or pager.

There is also a posibility to kick off fixit jobs triggered by these events. (Haven't tested this, and I'm not sure if it's desirable).

This might be a little off the track but just a thought anyway.

Andreas

[Paul R. Dittrich]

We put all our app developers on the standby node. The programmers get a protected playpen with the understanding that they get kicked off if a failover happens but they also give "real-time" feedback on performance and system health.

[Todd Legere]

Thanks to all. I am implementing this week end. Will assign points at end of this week.