lost quorum, pv not accessible... Reestablished quorum. PVlink recovered. What Happened?

[Ringoot]

I found these worrying messages in our syslog:
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: Lost quorum.
Dec 3 22:22:02 illuin.oma.be vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:
Dec 3 22:22:02 illuin.oma.be vmunix: <31 0x030400>
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: PVLink 31 0x030400 Failed! The PV is not accessible.
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: Lost quorum.
Dec 3 22:22:02 illuin.oma.be vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:
Dec 3 22:22:02 illuin.oma.be vmunix: <31 0x030400>
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: PVLink 31 0x030400 Failed! The PV is not accessible.
Dec 3 22:23:44 illuin.oma.be vmunix: LVM: VG 64 0x040000: Reestablished quorum.
Dec 3 22:23:44 illuin.oma.be vmunix: LVM: VG 64 0x040000: PVLink 31 0x030400 Recovered.
Dec 3 22:23:44 illuin.oma.be vmunix: LVM: VG 64 0x040000: Reestablished quorum.
Dec 3 22:23:44 illuin.oma.be vmunix: LVM: VG 64 0x040000: PVLink 31 0x030400 Recovered.

But there was no apparent service disruption.

From what I found on the forum of HP, this means that either the disk or the path to the disk failed.
However everything appears fine, never experienced disruption and no further errors found.

In the fora there is often suggested to use cstm or xstm to check all devices

I tried (currdevstatus (cds)) cds in cstm but got this back:
cstm>cds
^-- (CurrDevStatus) is currently disabled. --

cstm>

in xstm I see all disks green except these:
transtec T6100S12R1-E
0/1/1/1.0.2
0/1/1/1.0.3
0/1/1/1.0.4

they are yellow and marked: "incomplete information"

Haven't found this number : PVLink 31 0x030400 back there.


Thanks,

Joost

[R.K. #]

Hello,
Find the VG and disk involved:

0x040000 --> VG
# ll /dev/*/group | grep 040000 | awk '{print $10}'

0x030400 --> disk
# ll /dev/dsk | grep 030400 | awk '{print $10}'

[D Block 2]

how many PVs make up the VG ? Thinking there might just be one PV ?

Are there more than 1 path to this PV ? Check out running: vgdisplay on the /dev/vgX where the vgX is the VG name. you can post the output here if you like.

[sujit kumar singh]

Hi

As per these informations:

Dec 3 22:22:02 illuin.oma.be vmunix: <31 0x030400>
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: PVLink 31 0x030400 Failed! The PV is not accessible.
Dec 3 22:22:02 illuin.oma.be vmunix: LVM: VG 64 0x040000: Lost quorum


do the following


cd /dev/dsk
ll | grep "030400"
get the /dev/dsk/cXtydZ

pvdisplay -v /dev/dsk/cXtYdZ

also

ll /dev/*/group | grep -i 0x040000

get the VG name like

then
vgdisplay -v /dev/


please post the O/ps

Regards
sujit

[sujit kumar singh]

If that is a path going offline to the disk, you can see by ading the alternate paths for that LUN using the vgextend and so on.

In that case if one of the path to that LUN fails the LUN is accesed by the other path and so on.

Also need to check from the FC side SAN what is causing that to be offline , Might be the FC card / SAN Swicth port / Cable causing some issues.

Regards
sujit

[Ringoot]

thanks for the suggestions.
Apparently it is a disk in an external transtec disk cabinet that it is referring to.
Though all the disks there have just green leds, no error conditions displayed on the LCD screen either.

Until now the error has not reappeared in the syslog.

I will look up tomorrow (it's late now) if there is any console of this transtec box and how to access it to see if it would give me more info.