lvm frustrating problem

[Wessam]

Hi,

(on hpux 11.00)
I am facing a very frustrating problem, here is the scenario:

a currently activated vg with one lv mounted and another unmounted.
lv1 (mounted) exists on 2 PVs (pv1 and pv2)
lv2 (unmounted) exists on 5 other PVs (pv3, pv4, pv5, pv6, pv7).
now pv3-pv7 become unavailable (disks fail). this does not corrupt my currently mounted lv1 since it resides on different pvs (pv1, pv2)
i would like now to remove lv2 and the unavailable pvs (pv3-pv7).

this is almost immpossible.. i tried several ways to do it.
vg export and import wont work since it reads lvm metadata from pv1 and pv2 and believes the VG should have 7 PVs instead of 2 and wont even vgchange (even with -q n, since quorum is gone !)
the failed disks will not be replaced, so there is no way to vgcfgrestore anything..
please replay ASAP, as this makes the system unstable (next reboot will not mount lv1 since vg wont activate) !

[Paul_481]

Hi Wessam,

Please peform "vgscan -a -v" this will recreate the lvmtab therefor removing the failed disk.

Don't forget to rename first the the old lvmtab "mv /etc/lvmtab /etc/lvmtab.old".

Regards,
Paul

[Wessam]

already done that, this wont solve the problem since vgscan will rebuild only vgs in which all pv exist.
if you have a vg with 5 pv which dont exist, vgscan reports a problem on that and wont enter it in lvmtab

[Girish_17]

Hi,

1. Use "lvremove" to remove lv2
2. Use "vgreduce" to remove (pv3, pv4, pv5, pv6, pv7) from vg

This should work.

Regards,

[Wessam]

lvremove does not work since it must query the pvs first to remove and lv.
there is no way in lvm to force a removal of an lv if its pvs cannot be queried

[Girish_17]

So what about vgreduce

[Wessam]

vgreduce wont vgreduce since it reports that those pvs still contain extents of lv2.
it asks to first remove the lvs in order to remove the pvs

[Robert-Jan Goossens]

Wessam,

Have you got enough space to temp copy lv1 to another VG ?

Robert-Jan

[Wessam]

i would if i could..
this filesystem contains around 3+ million files
it took about 5 days last time just to copy.. and i had to stop cp in the end because i got tired. the only way was to lv mirror.

[Robert-Jan Goossens]

did you try the vgreduce -f option ?

[Wessam]

vgreduce -f reports that pv blah blah still has extents existing on it from lv blah blah
and then repotrs a bunch of errors on each unavailable pv like this:
-------------------
vgreduce: Couldn't query physical volume "/dev/dsk/c24t5d0":
The specified path does not correspond to physical volume attached to this volume group
Not all extents are free. i.e. Out of 953 PEs, only 857 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1
Here's the map of used PEs

--- Logical extents ---
LE LV PE Status 1
1728 lvol1 0000 ???
1729 lvol1 0001 ???
1730 lvol1 0002 ???
-------------------

the list goes on for each PV that is unavailable to the vg.
ofcourse, lvreduce does not work since lv2 now is reported as 0 extents (but still cant remove any pvs because lvm thinks they are still being used, but dont know who is using them - as reported in the map above)
lvremove does not work either cause this is the error reported:

-------------------
lvremove: Couldn't query physical volume "/dev/dsk/c23t5d0":
The specified path does not correspond to physical volume attached to this volume group
-------------------

it is a bit of a dangle.. since this filesystem in online now and i cant interrupt users.

[Robert-Jan Goossens]

I'm out of of online options. The fastest methode could be to reboot your system in single user mode, mount /usr filesytem and lvreduce lvremove the effected lvol/disks.

Robert-Jan

[Wessam]

ok i think i got how to fix the problem:
here is the steps:

1) remove current lvmtab
2) vgscan -av to rebuild lvmtab without the bad vg
3) vgimport to a new vg with only the two functioning PVs (pv1 and pv2)

4) activate the VG without quorum (vgchange -a y -q n vgname)

5) vgdisplay will report only the two functioning PVs as its members, and will report all LVs (including the missing PVs LV - lv2)
6) lvremove the bad LV (lv2)

this seems to work online (without unmounting and deactivating the vg). i will try to actually do that at night and ill report back updates.

[Wessam]

thanks to all of you for your contributions
robert-jan: i think this would work but i'll try my solution first and i'll let all of you know how it turns out.

its frustrating tha lvm has no way to force-remove a PV or LV its its corresponding devices cant be queried !

i think HP should patch that in the next LVM pathes.

[Dean Johnson_10]

Take a look at lvreduce with the -k option.
You need to check the pv key using lvdisplay
on lv2
Here is an example of one I have done previously (although this was on a failed mirror)

# lvdisplay -v /dev/vg00/lvol6
--- Logical volumes ---
LV Name /dev/vg00/lvol6
VG Name /dev/vg00
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 80
Current LE 20
Allocated PE 40
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t8d0 20 20

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
0000 /dev/dsk/c0t8d0 0202 current ??? 0202 stale
0001 /dev/dsk/c0t8d0 0203 current ??? 0203 stale
0002 /dev/dsk/c0t8d0 0204 current ??? 0204 stale
0003 /dev/dsk/c0t8d0 0205 current ??? 0205 stale

etc
In this instance the failed disk is PV2
So to remove used

lvreduce -m 0 -k 2 /dev/vg00/lvol6

(NOTE - if you are not mirrored you do not require the -m 0)

Logical volume "/dev/vg00/lvol6" has been successfully reduced.
Warning: Can not determine all Physical Volumes on which mirrored copies of
the Logical Volume are located. "/etc/lvmtab" is missing Physical Volumes.
Warning: Can not determine all Physical Volumes on which mirrored copies of
the Logical Volume are located. "/etc/lvmtab" is missing Physical Volumes.
Warning: Can not determine all Physical Volumes on which mirrored copies of
the Logical Volume are located. "/etc/lvmtab" is missing Physical Volumes.
vgcfgbackup: /etc/lvmtab is out of date with the running kernel:Kernel indicates 3 disks for "/dev/vg00"; /etc/lvmtab has 2 disks.
Cannot proceed with backup.

This then showed lvdisplay as

lvdisplay -v /dev/vg00/lvol6
--- Logical volumes ---
LV Name /dev/vg00/lvol6
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 80
Current LE 20
Allocated PE 20
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t8d0 20 20

--- Logical extents ---
LE PV1 PE1 Status 1
0000 /dev/dsk/c0t8d0 0202 current
0001 /dev/dsk/c0t8d0 0203 current
0002 /dev/dsk/c0t8d0 0204 current

You may need to repeat for all PV's

Hope this helps

Dean

[Henk Geurts]

Dean is right.
lvm has the option to remove ghost disks...
lvreduce -k ....

see also thread ;
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=757397

good luck.
regards.

[Robert-Jan Goossens]

Dean/Henk,

Yes -k option works great if you have mirrored disks, but what if you are not using mirrors :-) lv2 (unmounted) exists on 5 other PVs (pv3, pv4, pv5, pv6, pv7).

Best regards,
Robert-Jan

[Dietmar Konermann]

Wessam,

just my two cents... if you are dealing with VGs that contain unavailable PVs then it's wise to specify "-A n" on the command line, since it's usually the (automatic) vgcfgbackup that gets stuck.

So in your original scenario lvremove -A n ... and then vgreduce -A n ... should have worked. After removing the PVs from the VG it's time to perform a vgcfgbackup manually, of course.

BTW, with the -k option you can specify a PV using its key instead of its path. This is useful for unattached PVs, but does not change anything else.

Best regards...
Dietmar.

[Dean Johnson_10]

Robert-Jan

You are correct - think I should have read the man page first !! :-)

Regards

Dean

[Wessam]

thanks for all, i have been enlightened.
just some comments...
lvreduce -k does not work unless you have mirrors.
if my situation, the unavail. PVs could not be removed.

I tried the scenario i presented earlier. I unmounted the good lv and unactivated the VG
exported and imported back using only the working PVs (pv1 and pv2)
it worked with problems.. except that i get errors that
"the kernel reports 7 PVs but you actually have 2"

the VG works fine now. but i did not know about the ability to use the PV key.. could someone let me know how to use this ? anyone has any resources on obtaining PV keys ?

I would also like to . somehow.. tell the two current PVs (by altering the metadata or something) that they are in a VG which contain only 2 PVs.. not 7 (just to avoid that annoying error msg).

the autobackup option: i did not disable it first but then i tried that two.. i dont think it would have made any difference.. the failed PVs were persistent in that VG and i did not find any way to remove them online. had to export and import to fix that.

still my question stands.. any ideas on how to force remove LVs or PVs from a VG if they have failed.. ALL ONLINE ?