Errors on Syslog , LVM: Lost quorum in vg[47]

[Bernard Bravo]

Hi,

Some expert with EMS Storage background could me on this.

Below is the contents of the syslog.log while starting up the HP V2600 server.

thanks in advance.
bernard

Aug 24 22:08:16 CCBDRP syslogd: restart
Aug 24 22:08:16 CCBDRP vmunix: disk H/W path 0/2/0.1.20.0.0.6.4 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[47], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [4] to disk H/W path 2/1/0.1.16.0.0.3.7 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[60], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 1/2/0.1.20.0.0.4.2 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[61], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [1] to disk H/W path 1/2/0.1.20.0.0.6.1 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[62], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 2/1/0.1.16.0.0.0.1 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[63], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [1] to disk H/W path 3/0/0.1.24.0.0.1.0 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[11], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [1] to disk H/W path 2/1/0.1.16.0.0.8.5 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[37], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 1/2/0.1.20.0.0.8.1 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[36], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 3/0/0.1.24.0.0.8.0 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[58], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [4] to disk H/W path 3/1/0.1.16.0.0.0.5 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[30], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 3/1/0.1.16.0.0.6.7 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[46], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 2/1/0.1.16.0.0.1.5 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[29], too many PVs missing!
Aug 24 22:08:16 CCBDRP vmunix: Error writing VGDA [3] to disk H/W path 3/1/0.1.16.0.0.1.4 (error = 5)
Aug 24 22:08:16 CCBDRP vmunix: LVM: Lost quorum in vg[31], too many PVs missing!
Aug 24 22:08:18 CCBDRP nettl[1744]: nettl starting up.
Aug 24 22:12:31 CCBDRP rpcbind: check_netconfig: Found CLTS loopback transport
Aug 24 22:12:31 CCBDRP rpcbind: check_netconfig: Found COTS loopback transport
Aug 24 22:12:31 CCBDRP rpcbind: check_netconfig: Found COTS ORD loopback transport
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: check binding for udp
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: check binding for tcp
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: check binding for ticlts
Aug 24 22:12:31 CCBDRP rpcbind: svc_tli_create: t_getname failed; T_IDLE
Aug 24 22:12:31 CCBDRP rpcbind: svc_dg_create: could not get transport information
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: Could not create rmtcall fd for ticlts
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: check binding for ticots
Aug 24 22:12:31 CCBDRP rpcbind: init_transport: check binding for ticotsord
Aug 24 22:12:34 CCBDRP inetd[8383]: Reading configuration
Aug 24 22:12:34 CCBDRP inetd[8383]: protocol = tcp
Aug 24 22:12:34 CCBDRP inetd[8383]: ftp/tcp: Added service, server /usr/lbin/ftpd.........

[Sridhar Bhaskarla]

Looks like you have some problem with your storage system.

Run ioscan -f > ioscan.out and verify the file to see if you are seeing any devices with NO_HW entry. As this is occuring on multiple paths, I would suspect the storage system than the server itself. Check the storage system or the physical connectivity ASAP.

What kind of the external storage you are using?.

-Sri

[Bernard Bravo]

Hi,

Its EMC ...

tnx

[Sridhar Bhaskarla]

I don't know how to check EMC. Seeing so many messages, it doesn't appear that the drives are bad. It should be a connectivity issue. Please check with your EMC support people to see if there is anything changed on EMC>

-Sri

[linuxfan]

Hi Bernard,

How are you connecting to your emc, direct connect through fibre? going through a hub or a switch?

Also if it is a fibre connection, what kind of fibre channel card do you have on your V2600 server?

There were multiple instances in our case when using the old FC card(can't remember the card number) but had to replace them with the newer FC card(This problem was mostly on L's and N's).

In your case looks like this problem occured when the machine was rebooted. Also do you have multiple connections to the EMC? Was any work being done on the EMC?

You normally get lost quorum messages when all the disks in the VG cannot be reached. Do a ioscan -fnC disk and make sure you can see all the disks. Normally with EMC disks another utility which helps is inq (latest version is 4.3.1.0). You can get it at
ftp://ftp.emc.com/pub/symm3000/inquiry/v4.3.1.0/

-HTH
Ramesh