LVM issue with SAN attached storage

[C W_1]

I had several volume groups on a box (superdome hpux 11.11) all SAN attached storage that gave me the "Warning: couldn't query physical volume" on a vgdisplay. This was solved by doint a vgchange -a y vgXX. But the weird thing is that the vg's were already up and running and activated. I'm just not sure why this happened, what caused it, etc.

I have another issue see the example with vg12 below. It gives the "Cross-device link" error on a vgdisplay. The vg is up and running, I am hesitant to reboot or restart the vg due to this error. See errors from lvm.11 program (tool I got from HP supprot) as well.



BUBBA:/:# vgchange -a y vg12

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c11t2d3":

Cross-device link

vgchange: Cross-device link

Volume group "vg12" has been successfully changed.



BUBBA:/:# ./lvm.11 -d /dev/rdsk/c11t2d3

Primary lvmrec has a bad magic trying secondary

Both lvmrecs of /dev/rdsk/c11t2d3 have bad magics. Failure!

BUBBA:/:# vgdisplay -v vg14

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c11t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c12t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query all of the physical volumes.

--- Volume groups ---

VG Name /dev/vg14

VG Write Access read/write

VG Status available

Max LV 255

Cur LV 1

Open LV 1

Max PV 16

Cur PV 1

Act PV 1

Max PE per PV 65535

VGDA 2

PE Size (Mbytes) 16

Total PE 1279

Alloc PE 1279

Free PE 0

Total PVG 0

Total Spare PVs 0

Total Spare PVs in use 0



vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c11t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c12t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query all of the physical volumes.

--- Logical volumes ---

LV Name /dev/vg14/lvol1

LV Status available/syncd

LV Size (Mbytes) 20464

Current LE 1279

Allocated PE 1279

Used PV 1





--- Physical volumes ---

PV Name /dev/dsk/c5t2d1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c11t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c12t2d1":

The specified path does not correspond to physical volume attached to

this volume group

vgdisplay: Warning: couldn't query all of the physical volumes.

PV Name /dev/dsk/c6t2d1 Alternate Link

PV Status available

Total PE 1279

Free PE 0

Autoswitch On





Bubba:/:# vgchange -a y vg14

Volume group "vg14" has been successfully changed.



BUBBA:/:# vgdisplay -v vg14

--- Volume groups ---

VG Name /dev/vg14

VG Write Access read/write

VG Status available

Max LV 255

Cur LV 1

Open LV 1

Max PV 16

Cur PV 1

Act PV 1

Max PE per PV 65535

VGDA 2

PE Size (Mbytes) 16

Total PE 1279

Alloc PE 1279

Free PE 0

Total PVG 0

Total Spare PVs 0

Total Spare PVs in use 0



--- Logical volumes ---

LV Name /dev/vg14/lvol1

LV Status available/syncd

LV Size (Mbytes) 20464

Current LE 1279

Allocated PE 1279

Used PV 1





--- Physical volumes ---

PV Name /dev/dsk/c5t2d1

PV Name /dev/dsk/c11t2d1 Alternate Link

PV Name /dev/dsk/c6t2d1 Alternate Link

PV Name /dev/dsk/c12t2d1 Alternate Link

PV Status available

Total PE 1279

Free PE 0

Autoswitch On

[Sheriff Andy]

Just out of curiosity, did you have a hiccup on your SAN? Any HBA connection go down, brocade, controller on your SAN go down?

Looks to me like it was more of a problem with your SAN.

[C W_1]

It's possible that something hiccuped on the SAN, I have no visibility into that. And I don't really know when it happened so I can't get much from our SAN group.

Still, I don't think it explains the particular problem with vg12. What is a "Cross-device link"? Leads me to believe there is some problem with the data (although it is mounted and functioning...) or the LVM header..


BUBBA:/:# vgchange -a y vg12

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c11t2d3":

Cross-device link

vgchange: Cross-device link

Volume group "vg12" has been successfully changed.

[Sameer_Nirmal]

Tried to assemble to scenario which you might have ( correct me if I am wrong ) .
All the those VGs were mounted and working you are running vgdisply on them and getting all those errors .

I would advise cross check everything before doing any changes or running comamands as it maybe disasterous.

Cross-Device link means the device is being owned by more than one VG. The cause of this crossing is the mismatch between lvmtab and LVM headers on the disk.
check if the disk belogs to more than one VG in the lvmtab. If it does, then lvmtab ncould be recreated using vgscan.
I guess there won't be such mismatch in your case since you already used lvm tool and if we go as per the errors given for VG12, the LVMREC ( both records ) of PVRA or PVRA itself of c11t2d3 are corrupted.
In this case , the LVM headers need to be restored from /etc/lvmconf/vg12.conf
Check if has the disk entry in it using # vgcfgrestore -l -n vg12
If it does then the LVM header could be restored on the disk using
#vgcfgrestore -n vg12 /dev/dsk/c11t2d3
#vgchange -a y /dev/vg12

Note there maybe misleading case as the VG maybe pointing to wrong PV itself ( because of issues with SAN storage ) and hence the lvm tool is couldn't find the lvmrec. You need to cross check all using all previous configuration information which you might have noted like VGs , their PVs , LVs etc..
I suspect the corruption could be caused by problems at SAN storage side. Which SAN Storage is being used?

For VG14 issue , I suspect the cause as connection throught HBA to the LUN which is getting down ( may be intermittenty ) causing those alternate link failures.

Once again verify all before doing any actions

[C W_1]

Sameer, thanks. vg12 has the disk in vg12.conf. I figured at some point I would be trying to restore the LVM headers. When I get downtime tonight I will try and restore them.

#vgcfgrestore -l -n vg12
Volume Group Configuration information in "/etc/lvmconf/vg12.conf"
VG Name /dev/vg12
---- Physical volumes : 2 ----
/dev/rdsk/c11t2d3 (Non-bootable)
/dev/rdsk/c12t2d3 (Non-bootable)

I am really curious as to how I got into this state, however. I know a pvcreate can blow away headers, but I'm pretty sure that isn't what happened here. Can some event on the SAN side cause this (it's Hitachi by the way)? If so, I wonder if there is something in syslog I can monitor..

[C W_1]

vgcfgrestore on vg12 gave "Invalid LVMREC on Physical Volume "

so i decided to:

pvcreate /dev/rdsk/c11t2d3

at this point vgcfgrestore complains about no vg12 in existence...so I run:

vgscan -av
vgchange -a y vg12

now finally vgcfgrestore works and vg12 is happy again.

I am still curious...could the something on the SAN side cause this?

[Sameer_Nirmal]

I suspect the LVMRECs of that disk has been accidently over-written. To know the cause of this , one need to know the LVM configuration with the Hitachi Storage LUNs, any pathing s/w being configured etc.