Missing cluster lock

[Dave Forber]

Hi

We've been having problems over the last few days with our Oracle app running on a Service Guard cluster over two RP8400's. I'm getting quite a lot of error messages in syslog.log, but the main gist seems to be the following:

Jan 5 13:19:45 gcwsdb01 cmcld: WARNING: Cluster lock on disk /dev/dsk/c5t13d5 is missing!
Jan 5 13:19:45 gcwsdb01 cmcld: Until it is fixed, a single failure could cause all nodes in the cluster to crash

Not (yet) trained in the mysterious arts of ServiceGuard, can anyone point me in the right direction?

Thanks.

[melvyn burnard]

your Serviceguard configuration is set up to use the disc c5t13d5 as a lock or tie breaker disc. This node is currently indicating it cannot talk to the disc, or the cluster lock setting has been removed from the physical disc.
The system will check for this disc every 60 minutes so you are probably seeing this message every 60 minutes.

Does the other node have the same message in syslog? if so, the disc has a problem, if not then you may have lost a link to the PV from this node.

[Geoff Wild]

It is telling you that when MC/SG was configured, the cluster lock disk was: /dev/dsk/c5t13d5

You can generate a new ascii file - and it will tell you what the cluster lock should be:

cmquerycl -v -C /etc/cmcluster/new.ascii -n node1 -n node2

where node1 and 2 are the server names of the nodes in your cluster.


Look at the file for:

FIRST_CLUSTER_LOCK_VG

and

FIRST_CLUSTER_LOCK_PV

You will then need to add to you existing ascii file...or just use the new one (edit it first! - set: CLUSTER_NAME

as well as add your vg's to the end:

Example:

VOLUME_GROUP /dev/vg50
VOLUME_GROUP /dev/vg51
VOLUME_GROUP /dev/vg55
VOLUME_GROUP /dev/vg56
VOLUME_GROUP /dev/vg57
VOLUME_GROUP /dev/vg52

Read Managing MC/SG for your version:

http://docs.hp.com/en/ha.html#ServiceGuard


Rgds...Geoff

[Dave Forber]

I should have probably also informed it is running HP-UX 11i with:

MC/ServiceGuard NFS Toolkit A.11.11.0.2
ServiceGuard OPS Edition Bundle A.11.14
HP Cluster Object Manager B.01.04
ServiceGuard Manager A.02.00

Cheers.

[Stephen Doud]

If you confirm that the disk at c5t13d5 (on gcwsdb01) is in good shape, pull the "cminit.tar.shar" file from this site:

ftp://contrib:9unsupp8@hprc.external.hp.com/crash

Place it on the affected server(s) and unpack it:
# sh cminit.tar.shar

Two files will unpack; an executable (cminitlock) and a README.

Read README for an explanation on how to use the command to either view the cluster lock status or restore the cluster lock structures to the disk WHILE THE CLUSTER IS RUNNING.

If this doesn't solve the problem (wait up to 60 minutes and inspect syslog.log to see if the WARNING message has no printed) - then perhaps the disk is not functioning properly or it is not supported as a cluster lock disk... or some other random thing.

-SD

[Robert C. Dunavent]

The FTP site idenitifed above requires a username/password. Anyone know what it is?

[Dennis Handly]

>The FTP site identified above requires a username/password.

You'll probably need to contact the Response Center.  That site may not even have that tool anymore, since this post is so old.