WARNING: Cluster lock failure.

[Nischal Tiwari]

Hi

I am facing an error with the serviceguard two node cluster.
OS:11.23
serviceguard:11.16
Error i get in syslog is
cmcld: WARNING: Cluster lock on disk /dev/dsk/c4t0d0 is missing!
cmcld: Until it is fixed, a single failure could
cmcld: cause all nodes in the cluster to crash

[sujit kumar singh]

hi
This is a serious case and you need to confirm the failure of the lock disk as being shown in the syslog and need to replace the disk immediately.

view the cluster config ASCII file for the lock VG and lock PV entries and that ensure that it is the same as the disk as being reported.
Try doing the basic tests on the disk to see if that is available.
for the Cluster lock VG as you can see in the cluster ASCII file say vgXX, do by seeing
vgdisplay -v vgXX
and look that the disk is Available.

see the disk as CLAIMED in ioscan -fnCdisk.

do a dd command to read few blocks on the disk.

dd if=/dev/dsk/cXtYdZ of=/dev/null count=1000

do a

pvdisplay -v /dev/dsk/cXtYdZ


these all shall help you confirm if the disk has failed,

then go on to replace the disk getting the cluster down and take a downtime of all the packages and the cluster if you can feel like that the disk is to be repalced.


NOTE: An SG Cluster shall keep up and running with all the packages even if the cluster lock disk fails but in the lock disk failed condituion if the Cluster reforms somehow like reboot of any node of the cluster then the results shall be entirely unpredictable and the Split brain Syndrome can occur.

please take this as very seriously if reqd log a case immediately with HP.
regards
sujit

[Johnson Punniyalingam]

Serviceguard (on each node) tests the cluster lock structure once an hour. Since one server is not reporting the errors, but the other server is - it would appear that the server reporting the errors cannot perform I/O on the disk.
See if the system can read from the disk:
# dd if=/dev/dsk/c4t14d1 of=/dev/null bs=64k
If this succeeds, perform 'cmviewconf' on the node reporting the error and confirm that the lock disk path on that node is correct.
Also consider performing
# cksum /etc/cmcluster/cmclconfig - on both nodes and confirm the numbers match across nodes.

Thanks,
Johnson

[Nischal Tiwari]

Hi Surjit,

I have done the checks which you have suggested and disk appears to be ok.


Hi Johnson,

I have used the dd command and verified the disk.All ok.
checksum reports same value.Lock disk path value ok.



Regards,
Nischal

[sujit kumar singh]

Hi
As suggested by Johnson, please run the cmviewconf from the node on which you are getting the syslog error and please post the O/P.

As you are able to see that the disk is alive in ioscan and vgdisplay, pvdisplay and the dd success shows that the disk is available?

Is that disk also having data on that?

Please check the syslog of the other node and also see if there are some error messages in there.
On the faulty node are you still getting the error messages? Just to make sure that the syslog error was transient in nature or not?

This can be caused due to some link failure of the disk with the cluster node or might have been removed or also can be due to the Lock VG Data corruption on the disk.



http://docs.hp.com/en/B3936-90122/ch07s04.html#cegecefc

Have a look for replacing the Lock Disk in the Cluster


http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1236585672039+28353475&threadId=1182812

regarding the cminitlock tool you can have a look at the above thread and see if you can get that requesting the HP Support.
regards
sujit

[Avinash20]

If you are sure, that the disk is good and the disk is indeed a Cluster lock disk as mentioned in cluster configuration, then you need to use cminitlock tools

The tool will re-initialize the disc that is accessed by the device file listed in the cluster ascii and binary

usage: cminitlock [-v] [-t] vg_name pv_name
-t Test the cluster lock only.
-v Verbose output.

This command will initialize a cluster.
lock disk and then query the disk to
validate the disk was initialize
successfully. If the -t option is specified,
the cluster lock is only queried.

You need to get this tool from HP.

[Eric SAUBIGNAC]

Bonjour Nischal,

What about a simple "cmapplyconf -C YourClusterConfFile" ? cmapplyconf is supposed to init properly cluster lock disk.

Eric

[Stephen Doud]

I agree with Avinash - it sounds like you need to reinitialize the cluster lock structure as long as the cluster lock disk identified by the message is still the correct disk. The cminitlock utility can be acquired from the HP Support Center.

[Vinod Kumar Yadav]

Hi Team,

I Face the same issue in My Environment 2 or 3 times.

Take Down Time from Customer.
Frist check the disk & vg status & perform the below steps.

1.Take the backup of cluster.ascii file.
2.Down the pkgs running on both node
3.cmcheckconf -v -C /etc/cmcluster/cluster.ascii
4.cmapplyconf -v -C cluster.ascii
5.Start the pkg & check
6.Check the syslog the error will not come again.

Thanks

[Basheer_2]

Get the current running cluster config file

cmgetconf -v -C cluster-name > outfile
check your
FIRST_CLUSTER_LOCK-VG
FIRST_CLUSTER-LOCK-PV

vgdisplay -v lock-vg (repalce from the above)
pvdisplay -v lock-Pv (replace from the above)

[Roberto Arias]

Hello Nischal!!
jaja i have your problem with cluster lock disk too.
cminitlock is not supported tool but with it you can put the mark of cluster lock disk in your disk.

If you don't has network problem or the node of cluster not lost connectivity with other nodes you don`t hav any problem.

The best is reconf cluster with new lock disk. In my cluster I changed the cluster lock disk for other and now i'll stop the cluster for reconf the cluster.ascii and re-cmapplyconf
best regards