Cannot purge LVM meta data from unwieldy, defunct disk

[Ralph Grothe]

Hi,

this has probably been posted here many times, though I couldn't find any matching hits on searches in the technical knowledge base for keywords like "lvremove fails|hangs".

Problem is a broken disk (i.e. primary boot disk in vg00).

So far I managed to lvreduce all lvols in vg00 so that only the PEs on alternate mirror disk are in use anymore,
*except* for one lvol11 that unfortunately wasn't mirrored and was mounted.
Anyway, the data on this mount wasn't too valuable so that I don't mind any loss there.

# lvdisplay -k -v /dev/vg00/lvol11|head -20
--- Logical volumes ---
LV Name /dev/vg00/lvol11
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 52
Current LE 13
Allocated PE 13
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c1t2d0 13 13




Of course all access to that filesystem caused processes to hang why also various umounts failed.
I deleted the entry from /etc/mnttab, and now at least bdf doesn't hang.

I tried various lvremoves and lvreduce -l 0 but even with the force switch -f I don't get the LVM tidied up.

A colleague even pulled the disk from the machine slot while I afterwards retried all the lvremove, vgreduce etc.

How can I get rid of the LVM data for this defunct disk without having to reboot the machine?

Btw, between the various lvreduces I issued in order to reduce the mirror I always manually had to "rm /etc/lvmconf/lvm_lock" so that the next lvreduce would work at all.

Rgds.
Ralph

[Massimo Bianchi]

Hi,
i do not understand if the lvol still appears to be mounted.


In that case, i think that you can only issue a reboot.

Otherwise, try with

vgreduce -f vg00

this should remove missing disks from the information stored in lvm.

Massimo

[Ralph Grothe]

It's only these 13 PEs that are causing the trouble:

# pvdisplay -v /dev/dsk/c1t2d0|head -22
--- Physical volumes ---
PV Name /dev/dsk/c1t2d0
VG Name /dev/vg00
PV Status unavailable
Allocatable yes
VGDA 2
Cur LV 1
PE Size (Mbytes) 4
Total PE 4340
Free PE 4327
Allocated PE 13
Stale PE 0
IO Timeout (Seconds) default
Autoswitch On

--- Distribution of physical volume ---
LV Name LE of LV PE for LV
/dev/vg00/lvol11 13 13

--- Physical extents ---
PE Status LV LE
0000 free 0000

[Mark Grant]

I really wouldn't like to try this myself on vg00 but if the disk is completely broken, I'm wondering if re-naming /etc/lvmtab and doing a vgscan might do the trick. If the disk isn't completely broken then maybe overwriting the LVM data on the disk with dd and then doing the vgscan thing might do it.

I really don't know if this would work though or how dangerous it might be.

[Dietmar Konermann]

Ralph,

as long as the file system is mounted you don't need to worry about LVM... the lvol is open and thus cannot be removed. Your 1st task would be to umount the file system, and that is indeed not easy if the disk is not responding.

The IOs to that disk are retried "forever" in kernel land you have only 2 options to get this cleared, 1) get that IOs through/failed or 2) reboot.

Since no valueable data is on that lvol anyway you could try to replace the failed disk with another one now. Maybe you are lucky and the situation is cleared... i.e. the umount goes through.

Good luck...
Dietmar.

[Ralph Grothe]

Dietmar,

I would have tried another disk if we only had a spare one.

But I was lucky to discover that there is still a valid HW support contract for this server.

Thus I issued a support call to get the disk replaced.