MirrorDisk/UX: lvreduce hangs while attempting to remove failed disk.

[ricardor_1]

Hi there,

I´ve got a bad disk which is not in the NO_HW state. Most of the PEs are stale. We cannot reboot that machine now, so we chose to lvreduce each of the local filesystems so we can remove the disk and rsync the mirror.

We do not have the pvchange -a n patches.

When running lvreduce -m 0 /dev/vg00/lvolX /dev/dsk/ it returns the LV was successfuly reduced (and lvdisplay confirms that), but it hangs after some lvlnboot: Logical Volume has no extents.

What should be done? the syslog keeps throwing scsi errors...

Thank you.

[Sandman!]

What is the server model and what version of HPUX is it running and which command's hanging lvreduce or lvlnboot? Post the output of strings /etc/lvmtab and force out the bad PV by doing vgreduce -f vg00.

~cheers

[Ramesh S · Accepted solution]

Hi

Please post your strings /etc/lvmtab output and vgdisplay vg00 output.

As Sandman mentioned, try to pull out the PV from VG00 by "vgreduce -l -f vg00" and run "lvlnboot -R"

Thanks & Regards

Ramesh

[Mike Shilladay]

Hi Ricardor,

You could always try the lvreduce command with the 'k' switch.

For example to remove the extents on a specific disk, (in my example c5t0d0) you would use the command

# lvreduce -m 0 -k /dev/vg00/lvol5 /dev/dsk/c5t0d0

You will need to do this on all the logical volumes in the volume group and then remove from the disk from the volume group with the vgreduce command. You may need to force it with the f option as mentioned in previous post.

Hope that helps,

Mike.

[ricardor_1]

We're running 11.11. ps shows it' s actually the lvreduce command that hangs.

lvreduce -m 0 -k /dev/vg00/lvolX /dev/dsk/

fails complaining that the remaining disks do not have too the PEs to that LVOL, which is odd, since all the extents are current within the remaining PV.

I was able to kill -9 it, but it only returns after some 30-40 minutes. I suspect it might be the vgcfgbackup after the lvreduce that' s hanging? What about the -A n switch?

The overall procedure I'm trying is:

- vgexporting (preview mode) other VGs.
- lvreduceing all the filesystems on the mirror set
- vgreduceing the faulty PV with -f
- Remove the deffective HD
- Wait for NO_HW, then insert the new HD
- The bootsectors
- create the PV
- vgextend
- lvextend recreating the mirror.

I'm currently stuck at the second step... Any suggestions/corrections?

Thank you.

[ricardor_1]

The vgdisplay -v output (some LVs removed to save space..)

--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 11
Open LV 11
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 4384
VGDA 4
PE Size (Mbytes) 16
Total PE 8748
Alloc PE 7994
Free PE 754
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/stale
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 128
Used PV 2

(many other LVs, all stale... the following was the one I was able to lvreduce, but only after waiting 40 minutes the lvreduce command (being killed -9) returned).

LV Name /dev/vg00/lvol13
LV Status available/syncd
LV Size (Mbytes) 10304
Current LE 644
Allocated PE 644
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c2t0d0
PV Status available
Total PE 4374
Free PE 23
Autoswitch On

PV Name /dev/dsk/c2t1d0
PV Status unavailable
Total PE 4374
Free PE 731
Autoswitch On



The c2t2d0 HD is CLAIMED.

The meaning part of lvmtab:
/dev/vg00
/dev/dsk/c2t0d0
/dev/dsk/c2t1d0

[TwoProc]

Don't reduce it. Just replace the disk with the new one and run vgcfgrestore to the replaced disk, and then use vgsync.

This way, you won't get hung with this whole process.

[likid0]

Thats true, there is no need to take the disk out of the vg00, you can just swap the disk, and then restore the lvm header with the vgcfgrestore command.

If you want to reduce a disk that is not present on the system you need to use the -k option:

If the disk was not available at boot time (pvdisplay failed) then the lvreduce command fails with an error that it could not query the physical volume. You can still remove the mirror copy, but you must specify the physical volume key rather than the name. You can get the key using lvdisplay with the â k option as follows: # lvdisplay -v â k /dev/vg00/lvol1 â ¦ --- Logical extents --- LE PV1 PE1 Status 1 PV2 PE2 Status 2 00000 0 00000 stale 1 00000 current 00001 0 00001 stale 1 00001 current 00002 0 00002 stale 1 00002 current 00003 0 00003 stale 1 00003 current 00004 0 00004 stale 1 00004 current 00005 0 00005 stale 1 00005 current â ¦ Compare this output with the output of lvdisplay without â k, which you did to check the mirror status. The column that contained the failing disk (or â ???â ) now holds the key. For this example, the key is 0. Use this key with lvreduce as follows: # lvreduce -m 0 -A n â k /dev/vgname/lvname key (if you have a single mirror copy) or # lvreduce -m 1 -A n â k /dev/vgname/lvname key (if you have two mirror copies)

[likid0]

sorry fot that, you can't read anything:

If the disk was not available at boot time (pvdisplay failed) then the lvreduce command fails with an error that it could not query the physical volume. You can still remove the mirror copy, but you must specify the physical volume key rather than the name. You can get the key using lvdisplay with the â k option as follows:

# lvdisplay -v â k /dev/vg00/lvol1


Compare this output with the output of lvdisplay without â k, which you did to check the mirror status. The column that contained the failing disk (or â ???â ) now holds the key. For this example, the key is 0. Use this key with lvreduce as follows:


# lvreduce -m 0 -A n â k /dev/vgname/lvname key (if you have a single mirror copy)
# lvreduce -m 1 -A n â k /dev/vgname/lvname key (if you have two mirror copies)

[ricardor_1]

Hi,

Thank you all for your valuable tips, but I still have a question.

Since I tried to lvreduce some of the lvols, and also vgcfgbackup has failed due to the bad HD (it did not execute, at least no successful backup message was returned), what is the state of the .conf files that'll be used for the restore? It's probably not up to date, right?

[Sandman!]

The upto date state of the vg00.conf file does not matter as long as they contain the PVs that make it up including the bad one. Simply replace the bad PV mirror vg00 and you should be all set.

~cheers

[ricardor_1]

Thank you. I was able to remove the disk. vgcfgbackup did not work, I had to vgreduce it and reconstruct the mirror by lvextending each of the lvols again.

I still have a problem though. The lvols pertaining to / and /var do not sync one PE each one. It complains of I/O error, which I suspect is rather a data corruption problem than a physical compromise of the first HDD.

How can I solve that problem considering it´s the root filesystem that does not resync? I though of:

- booting in single user,
- creating a new lvol, copying / with tar
- exclude, piping to a tar extracting to a the new /,
- updating fstab on both lvols
- running lvlnboot updating the root lvol and then rebooting to multiuser again.

the same to the /var lvol.

Any other suggestions? Should this work?

[Mridul Shrivastava]

This will not work since it is a root filesystem.

You have to take backup using ignite ie make_tape_recovery if you have tape drive connected to the system. Then restore it. But do you really think if there is data corruption this will sove the issue ?? Since you will be restoring the same data back to the system ??

Rather I would suggest to restore the old backup if you have any?

One way to confirm the bad block would be to run read test using dd command. or you could check the same in cstm.

[ricardor_1]

Thank you for the tips. Actually I *think* it might be a data corruption problem because the "I/O error" mirrordisk complains of does not generate any hardware messages/kernel information.

Doing a make_net|tape_recovery would confirm if there´s any data loss, in the case there´s data on the stale PEs.

Our recovery tapes are rather outdated (~ 8 months) so I will probably try to restore a new recovery image. If it doesn´t work, then I´ll try the old ones.

Thank you again. If anyone has any other comments, I would like them to hear from you.

[Ben Dehner]

This is an old thread, but I had the same problem.  A failed disk in my mirrored root volume group.  'lvreduce' would hang, 'vgreduce', depending on the option, either did nothing or errored out with 'extents in use', and 'pvchange -a n' also did nothing useful.

Finally, I found the following approached worked:

important step:  first pull the bad disk and replace it with a working disk

now remove mirrors off of the bad path -- it will complain about 'invalid physical volume', but it won't hang on SCSI errors from the failed device.

lvreduce -m 0 <DISK_PATH>

pull the disk from the volume group

vgreduce vg00 <DISK_PATH>

create a physical volume on the replaced disk.  ( -B since this is a boot disk).

pvcreate -B <RDISK_PATH>

make the disk bootable 

mkboot <RDISK_PATH>

mkboot -a "hpux" <RDISK_PATH>

add the disk back to our volume group

vgextend vg00 <DISK_PATH>

re-establish mirroring for all LV's in the volume group

lvextend -m 1 /dev/vg00/lvolNN <DISK_PATH>

vgcfgrestore may perform many of the same steps.   I am not an LVM expert, but my primary concerns were that, since this disk is part of my root volume group, i) the disk had to be bootable with the LIF partition, and ii) as part of the root volume, the logical volumes had to exist *in the proper order* on the replaced disk if the system used this as the boot device.  Given my unfamliarity with LVM, I performed the manual steps that I knew would do what I needed.