Cluster lock device name changed after server crash

[peter carpenter_1]

I have a situation where a server which is part of a two node ServiceGuard cluster crashed and when it rebooted the device files names came up different. Therefore, it is unable to look at the cluster lock disk and outputs the following error in syslog.log:

cmcld: Unexpected error from cluster lock query (lock_id=0): Invalid argument.

When doing an spmgr display the unit is shown with the word Error where the hardware path should be. It looks like this was caused by the fact we are running Securepath 3.0d and did not make the device files persistent accross reboots. Therefore, I have been able to vgimport the cluster package units fine, but I have a call with HP raised regarding the cluster lock device.

My HP-UX guru has suggested we might need to remove the lun from spmgr and then re-attach it, ioscan and insf (assumingly on both nodes)
and then reconfigure the cluster with the new device name, which sounds feasible. I just wondered if anyone else has been faced with this problem? I'll post HP's response when I receive it.

[melvyn burnard]

If the device name has changed for the cluster lock disc, you will have to halt your cluster, make the relevant change to the cluster ascii file, then reapply teh cluster with the clusterlock disc VG activated on the node where you run the cmapplyconf.
I would also suggest this may be a good time to consider changing to a Quorum Server, rather than a cluster lock disc.
The software is free.

[freddy_21]

You must aware why your cluster crashed?

When the device changes, your cluster cant startup because at cluster ascii, you already state cluster lock device. if your device changes, your cluster cant find the cluster lock device. you must run vgscan, vgexport and vgimport for new devices.

Do you have configuration before and after changes? please compare instant number before and after.

Usually, i tried to make same with before changes. the command is ioinit. but you must know the original configuration.

At the future, collect ioinit at /etc/ioinit and /stand/ioinit. so if you get device changes after reboot, please copy ioinit file and put at /stand and /etc, and reboot again. I hope you will get the original devices.


Thanks
Freddy

[peter carpenter_1]

I guessed I would need to re-configure the cluster at some point, but the device name has only changed on one node (the one that crashed). So the primary node is still running fine with the original device file names. I wondered if anyone has had to deal with something like this before, and how straight forward it is to correct.

[peter carpenter_1]

Thanks so far guys, and just for info, the crash was due to a powercut in one of the server rooms. As the device files were not made persistent, the names changes upon start up.

[freddy_21]

1. create map file from primary node
vgexport -v -s -p -m mapvg* /dev/vg*
2. vgexport all share vg at secondary node
vgexport vg*
3. ftp map file from primary node to seconday node
4. create directory vg* at /dev
mkdir /dev/vg*
mknod /dev/vg*/group c 64 0x010000
note minor number ( 0x010000) is a unique number check with command: ll /dev/vg*/group
4. vgimport all share disk at secondary node
vgimport -v -s -m mapvg* /dev/vg*
5. change cluster ascii file for secondary node. with a new device
6. apply cluster ascii file with comand cmapplyconf -C
7 if success, you can startup your cluster


thanks
freddy

[John Bigg]

Just a point to note here since it conflicts with some info you already received.

A cluster will start without a cluster lock being available. This is true for both cluster lock disk and quorum server.

You cannot configure or re-configure a cluster without the lock being available, but if it becomes unavailable after the cluster is configured it does not prevent you starting the cluster. The lock is only required when a tie break situation arises. At this point if the cluster cannot communicate with the cluster lock then both halves of the cluster will fail.