LVM PV unavailable, ioscan and diskinfo OK

[Thierry D]

Hi all,

I am a bit confused, on one of my LV, I received these errors.

VM: VG 64 0x020000: PVLink 31 0x100000 Wrong Disk!
--> correspond to disk c16t0d0
ls -l /dev/* | grep 0x100000| grep 31
brw-r----- 1 bin sys 31 0x100000 Feb 5 17:33 c16t0d0

LVM: VG 64 0x020000: PVLink 31 0x120000 Wrong Disk!
--> correspond to disk c18t0d0 (Alternate Link to the first disk)
ls -l /dev/* | grep 0x120000| grep 31
brw-r----- 1 bin sys 31 0x120000 Feb 12 09:21 c18t0d0

LVM: VG 64 0x020000: Lost quorum.
This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:
<31 0x100000>
--> This one is "normal" since it is "unavailable".
--- Physical volumes ---
PV Name /dev/dsk/c16t0d0
PV Name /dev/dsk/c18t0d0 Alternate Link
PV Status unavailable


I tried a pvchange -a y of the disk but no vgcfgrestore because according to ioscan and to diskinfo all is good and no disk was replaced:

ioscan -fnC disk /dev/dsk/c16t0d0
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
disk 35 1/0/14/1/0/4/0.101.13.19.0.0.0 sdisk CLAIMED DEVICE HITACHI OPEN-V
/dev/dsk/c16t0d0 /dev/rdsk/c16t0d0
ioscan -fnC disk /dev/dsk/c18t0d0
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
disk 58 1/0/12/1/0/4/0.101.13.19.0.0.0 sdisk CLAIMED DEVICE HITACHI OPEN-V
/dev/dsk/c18t0d0 /dev/rdsk/c18t0d0

Also did a "dd" to /dev/null and no I/O errors.

My question is how is it possible LVM sees it as unavaible if the OS does see it correctly ?
How is it possible to fix it ....

uname -a
HP-UX XXX B.11.11 U 9000/800 1853849460 unlimited-user license

Regards,
Thierry

[Patrick Wallek]

Perhaps someone did a 'pvchange -a n' on the disk.

Try doing a 'pvchange -a y /dev/dsk/c16t0d0' and see what happens.

[johnsonpk · Accepted solution]

Hi Thierry,

Is that disk presented from SAN? if so check that same LUN is presented to some other server or not . I guess some one did a pvcreate -f on that disk :-(

check the VG ID of that disk and copare it with other pv in the same VG

to find out VG ID
echo "0x2008?4D" | adb /dev/dsk/c0t1d0

[Thierry D]

Thanks for the tips,

I alreday did tried the pvchange but without success.
But I understand now why, when I display the LVM header, it is "empty" ...


# echo "0d8208?DY" | adb /dev/dsk/c16t0d0
2010: 0 1970 Jan 1 01:00:00
# echo "0d8208?DY" | adb /dev/dsk/c18t0d0
2010: 0 1970 Jan 1 01:00:00

I am now checking with the SAN team if they did something regarding the visibility of the disk ...

Is there a way to recreate the LVM header without damaging the data after it ? (That would be better then recovering from backup)


Thierry

[Eric SAUBIGNAC]

Bonjour Thierry,

You can restore LVM metadata with 'vgcfgrestore -n VGName /dev/rdsk/c16t0d0'.

But I am afraid that if your LVM header is empty, the whole disk will also be emty :-(((

Eric

[johnsonpk]

Hi Thierry,
try this command and observe and compare out put
echo "0x2008?4D" | adb /dev/dsk/cxtydz

the output will look like below

2008: 894960601 1067429321 894960601 1067430042
this should interpret as

2008:

below is the conclusion that i got after comparing many servers those have shared and non shared volume groups so far (hp ux 11.11) (I dont have any proof )

CPU ID 1.. will be same for all PVS that are being created from same server

PVID will be uniq .disk path that has SAme PVID considered as alternate links

CPU ID 2 - will be same as CPU ID 1 if that PV is not shared ,(will be different in case of shared pv/vg )

VG ID - will be same for all pvs inside the VG . if the VG is shared across two servers, it will have same VGID



In your case compare CPU ID 1 and VGID with other PVS in the same VG ..I hope you can figure out waht ent wrong with that PV

[Thierry D]

Hi johnsonpk,

Unfortunately I wont be able to compare VGIDs from same VG with others PVs, there was only one disk ....

From the command about VGIDs, I got the following.... it confirms the fact the LVM header is "dead".

# echo "0x2008?4D" | adb /dev/dsk/c16t0d0
2008: -1016020538 1266339344 0 0
# echo "0x2008?4D" | adb /dev/dsk/c18t0d0
2008: -1016020538 1266339344 0 0

Still have to wait from the Storage team, but as Eric said, pretty sure I will have to recreate LVs and recover from backup ...

[johnsonpk]

Hi Thiery,
>> it confirms the fact the LVM header is "dead".

After seeing the output, it seems to me the LVM header is overwritten,,

# echo "0x2008?4D" | adb /dev/dsk/c16t0d0
2008: -1016020538 1266339344 0 0
# echo "0x2008?4D" | adb /dev/dsk/c18t0d0
2008: -1016020538 1266339344 0 0

Did you get any chance to see the same out put of vg00 disk..if the cpu-id for vg00 is also coming as -1016020538 ,then probably some one played from the same server itself and , SAN team might be innocent in that case :-)


BTW , what about dd on that disk ,does it report any error?

[johnsonpk]

also there is a chance that you will get the data back by vgcfgrestore if it was just pvcreate by mistake and no data was written to it

[Thierry D]

Finally all is good again !!

The issue came from the SAN config being changed and once restored back the disk was still not seen by LVM but with the following steps all went good :

fuser -cku /fs
vgchange -a n vgfs
vgcfgrestore -n vgfs /dev/rdsk/c16t0d0
vgchange -a y vgfs
fsck -F vxfs /dev/vgfs/lvfs
mount /fs

Thanks to all !
Thierry

[Thierry D]

see previous post