Missing cluster lock

[alexmartin]

Few months ago we lost a volume group which consist service guard cluster lock. when i realized i re-applied the cluster lock in to another volume group which serve a shared folder access to the users; and everything works perfectly as expected. afterwards i even did several shutdown for both nodes and and everything works fine. But since last few weeks i am getting some errors on both nodes when i execute 'dmesg' command and also showing errors in syslog as well. Please find the details at below and also in attachment. Actually the cluster package is now running on the secondary server; so, getting more warnings and erros on secondary server.

Alex martin


From primary node:

SCSI: Read error -- dev: b 31 0x057200, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000006c9ac000), from raw device 0x1f057200 (with priority: 0, and current f
lags: 0x40) to raw device 0x1f077200 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x010000: PVLink 31 0x057200 Failed! The PV is still accessible.
LVM: VG 64 0x010000: PVLink 31 0x057200 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000006c9ac000), from raw device 0x1f077200 (with priority: 1, and current f
lags: 0x0) to raw device 0x1f057200 (with
priority: 0, and current flags: 0x0).

From Secondary Node:
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000004c85c000), from raw device 0x1f055400 (with priority: 0, and current f
lags: 0x40) to raw device 0x1f075400 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x040000: PVLink 31 0x055400 Failed! The PV is still accessible.
LVM: VG 64 0x040000: PVLink 31 0x055400 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000004c85c000), from raw device 0x1f075400 (with priority: 1, and current f
lags: 0x0) to raw device 0x1f055400 (with priority: 0, and current flags: 0x0).
LVM: VG 64 0x040000: Lost quorum.
This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by
current link) must become available:
<31 0x055400>
LVM: VG 64 0x040000: Reestablished quorum.
LVM: VG 64 0x030000: Lost quorum.
This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by
current link) must become available:
<31 0x055300>
LVM: VG 64 0x030000: Reestablished quorum.

[melvyn burnard]

well from the above it looks like you are losing or have already lost a path to your shared storage, as the message says:
LVM: Performed a switch ......
and
PVLink 31 0x055400 Failed!

Investigate your storage links and verify those first.
It seems the link that is failing is the one down whoch you get to your cluster lock disc

[alexmartin]

Mr. Burnard,

How can i Investigate the storage links and verify those are working fine. I already used ioscan -fnC disk command and it shows all of them are CLAIMED. in addition to that i also checked all physical connections and LEDs on the storage and SAN switch, it looks everything fine.

Please see this error

Jul 10 23:22:06 noor_sec cmcld: Health query for cluster lock disk /dev/dsk/c5t5d3 shows BUSY lock
Jul 10 23:22:06 noor_sec cmcld: Check device, power, and cables.
Jul 11 01:01:39 noor_sec vmunix:

Thanks and regards,

[melvyn burnard]

I suggest you start by looking at any hardware logs, STM logs, logs on switches etc, and as the message says:
Jul 10 23:22:06 noor_sec cmcld: Check device, power, and cables.


Although if this all happend back in July, is it still valid information?