Forcing a cluster lock after the cluster is running

[Tim Ryan]

Hi Everyone,

I've got a two node cluster that seems to be working fine. Unfortunately, on last reboot neither system was able to obtain a cluster lock. Consquently, both systems now complain hourly with the following:
Dec 19 13:57:57 node1 cmcld: WARNING: Cluster lock on disk /dev/dsk/c1t8d6 is missing!
Dec 19 13:57:57 node1 cmcld: Until it is fixed, a single failure could
Dec 19 13:57:57 node1 cmcld: cause all nodes in the cluster to crash

The volume group that this disk (/dev/dsk/c1t8d6) is in is running fine in a package on node1. Also, I can mount the volume group up read-only on the other system and everything looks fine.

My question is, how do I get one node to get the cluster lock now that the cluster is up and running. I don't want to reboot.

OS - HP-UX 11.0
September Patches

Any ideas would be appreciated!
-Tim

[G. Vrijhoeven]

Hi,

this is from the manual:

http://docs.hp.com/hpux/onlinedocs/B3936-90053/B3936-90053.html

it states:

Replacing a failed lock disk mechanism is the same as replacing a data disk. If you are using a dedicated lock disk (one with no user data on it), then you need to issue only one LVM command, as in the following example:

# vgcfgrestore /dev/vg_lock /dev/dsk/c2t3d0
After doing this, wait at least an hour, then review the syslog file for a message showing that the lock disk is healthy again.

Hope this will help,

Gideon

[Sridhar Bhaskarla]

Hi Tim,

I would suggest to halt the cluster, delete configuration and apply the configuration. This will ensure that the cluster lock gets initialized properly. It is not advisable to do a vgcfgrestore if the disk is being used in the lock volume group for data.

Then you can start the cluster and the packages.

-Sri

[melvyn burnard · Accepted solution]

The previous answer will work ONLY if you performed a vgcfgbackup after the cluster lock disc was marked as such and the cluster lock BG was activated at the time.

If you do NOT have a valid vgcfgbackup file, then you have to either halt the cluster, activate the cluster lock vg and reapply the binary, or there is an unsupported utility , cminitlock, that you could log a call with your local HP RC and request. This utility will allow you to reapply the cluster lock info on line, but again, it is unsupported.

[Tim Ryan]

Where can I check to see if I have a valid vgconfigbackup file?

[Sridhar Bhaskarla]

Hi Tim,

As mentioned earlier, it's not advised to use vgcfgrestore. You can reapply the configuration safely (delete and apply) and it will take care of this issue.

For your information only, the backup files are under /etc/lvmconf directory as .conf files.

-Sri

[melvyn burnard]

You woul dneed to know whether you backed up the vg after creating the cluster lock disc config (applying the binary origionallY).

As mentioned previously, you could simply try using vgcfgrestore onto the the clster lock disc ONLY, and you would need to look at the files in /etc directory. The man page for vgcfgrestore will tell you the default directory, and also how to see if they have the correct pv info.
And it is quite ok to use the vgcfgrestore, unlike previous responses state.

[Tim Ryan]

Thanks for your help!

I did contact HP's Service Center and they gave me the cminitlock utility. I ran this and it immediately fixed the problem.

I would have done the vgcfgrestore, but my backup was the same as what I was currently using.

If anyone else has this problem. I strongly suggest getting the cminitlock utility from HP. It'll fix the problem in seconds!

Thanks again!
-Tim Ryan

[Stephen Doud]

The cminitlock tool - though not officially supported by HP (it hasn't been run through rigorous testing), has been found to be safe to use while the cluster is running.

The tool can be found at
ftp://contrib:9unsupp8@192.170.19.51/crash/cminit.tar.shar

Unpack it using:
# sh cminit.tar.shar
# tar -xvf cminit.tar

A README with instructions on how to use the tool is included.