2 failed disks, one in Primary, one in mirror

[N Ward]

I have 4 LVs each LV has made up of the same 4 disks.

Each PV is mirrored to its corresponding device.

c2t1d0--->c1t1d0
c2t2d0--->c1t2d0
c2t3d0--->c1t3d0
c2t4d0--->c1t4d0

In the primary c2t4d0 has failed, in the mirror c1t3d0 has failed. Currently the system is running fine using c1t4d0 along with c2t1d0,c2t2d0 and c2t3d0. The two failed sas disks are now in place and the HW target replacement (via sasmgr) has been done. I'm a little unsure how to proceed with the LVM work?

[Torsten.]

You should proceed with the vgcfgrestore command and sync the VG. Then check if you still have stale extends.

[Matti_Kurkela]

The important question is, is this a system VG (vg00) or not?

When replacing a failed disk in a mirrored system VG, you must pay special attention to the boot information. But if this is not a system VG, the procedure is pretty simple.
See the instructions for "Replacing a Mirrored Nonboot Disk" starting at page 31 of this document:

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad_WP.pdf

This document has good instructions for other scenarios too: you might want to keep it.

MK

[N Ward]

Hi thanks for your reply. The LVs are allocated out to VMs and are the VMs vg00. However I've been reliably informed this doesn't affect disk replacement, so no need to worry about boot stuff.

I've got that document and it doesn't deal with this scenario. 4 disks, 4 LVs and each of those disks in each of the LVs, with a disk failure on both sides, primary and mirror.

[Torsten.]

The scenario is not different to a single failed disk, as long as you don't loose the same data on both sides.

Just follow the documented procedure.

[N Ward]

Torsten thanks, I can do this with 'vgchange -a y' after the vgcfgrestore? I have done the vgcfgrestore on one disk and that was fine, just need to do the second disk. I have a feeling when replacing the first that I reduced the mirroring to allow me to replace the disk rather than using pvchange, will have to redo the mirroring?

PVGs are used with this volume group too, I think to keep the allocation to specific PVs. This is all a little grey area for me..

[Torsten.]

If you have done the pvchange -a n ... or vgreduce without updating the configuration files (... -A n ...), the vgcfgrestore restores the information about the mirrors.

In this case you probably need to do the vgchange -a y ... and check with vgdisplay, if all disks are available. A vgsync should be enough then.

[N Ward]

On the one disk I think I just did a straight forward vgreduce I didn't specifically say no cfg saving.

[Torsten.]

So if the LVOL is no longer mirrored, you need to mirror again. Check with lvdisplay.

[N Ward]

There is a .old file for the volume group in /etc/lvmconf. Could I not just rename this one (it contains the old disk) then run the vgcfgrestore?

[N Ward]

Also, yes, neither of the LVOLs are currently mirrored. This is the bit I'm unsure of because they are mirrored in a specific way, one specific disk to one specific disk. And each LV has the same 4 disks in it.

[N Ward]

There is a /etc/lvmpvg file. The disk I did the vgreduce on is missing. I believe I need to edit this file and put it back in. Then if I remirror, do I just specify all the disks on the command line? Would that match the PVs up correctly i.e.

c2t1d0--->c1t1d0 etc, etc.

[N Ward]

OK, this is where I am now.

/etc/lvmpvg file looks correct 4 disks in vmos_primary and 4 disks in vmos_mirror. I didn't edit the file, I added the missing disk (c2t4d0) using vgextend specifying the PVG to add to. One disk is showing as unavailable in vgdisplay -v vg_vmos. Its the disk that was removed via vgreduce (c2t4d0). I have done all the vgcfgrestore work and it went through ok.

No LVs are mirrored. If I mirror using the PVG name is LVM intelligent enough to know that one of the disks currently is use was from a previous mirror and not overwrite it with the new blank primary?

The other disk is a mirror that failed so no problem there.

[N Ward]

Anyone? I still don't see how the document can help in this scenario. I need to be sure that each PV mirrors to its corresponding counterpart.

No LVs are currently mirrored.. How do I mirror when one failed disk is from the mirror and one from the primary and I need to ensure that the PVGs are adhered to, with each disk mirroring to its counterpart?

If use -m1 for each LV do I specify the PVG name or the devices. If I specify devices what devices do I specify considering one of the mirror disks is in use already acting as the primary and will this cause the right PV to mirror to the right PV as in my original post.

If I mirror using the PVG name how will it handle the mirror currently being used as the primary?

[Torsten.]

Hard to sggust something without knowing the details.

Please attach

# vgdisplay -v
# lvdisplay -v
# strings /etc/lvmtab
# strings /etc/lvmpvg

[N Ward]

I appreciate the response, I will try and get these details. Due to the nature / security of the system it may take me until tomorrow to get the info off.

[N Ward]

Attached is the information requested. Bear in mind that as stated earlier, each device should be mirrored to its corresponding partner.

c2t1d0--->c1t1d0
c2t2d0--->c1t2d0
c2t3d0--->c1t3d0
c2t4d0--->c1t4d0

[Torsten.]

This is what we currently have:

PV Name /dev/dsk/c2t4d0
PV Status unavailable



So first we need to have this disk included again.


LV Name /dev/vg_vmos/lv_vm05_vg00
...
Mirror copies 0
...
Allocation PVG-strict/distributed
...
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c2t1d0 768 768
/dev/dsk/c2t2d0 768 768
/dev/dsk/c2t3d0 768 768
/dev/dsk/c1t4d0 768 768


All other LVOLS look the same.


So you first need to 4th disk back online.

[N Ward]

Have added in c2t4d0 now, its also a part of the /etc/lvmpvg file in the correct group and shows as available in the output from vgdisplay -v vg_vmos

I've just tried to mirror using "lvextend -m 1 /dev/vg_vmos_lv_vm05_vg00 vmos_mirror" and have got the error:

lvextend: Not enough free physical extends available.

Logical Volume "/dev/vg_vmos/lv_vm05_vg00" could not be extended

Failure possibly caused by PVG-Strict or Distributed allocation policies

[Torsten.]

This is expected, because now it is on

/dev/dsk/c2t1d0 768 768
/dev/dsk/c2t2d0 768 768
/dev/dsk/c2t3d0 768 768
/dev/dsk/c1t4d0 768 768

So I would disable PVGstrict, mirror to the remaining disks, remove the mirror from all disks in one group, enable PVG strict and mirror again.

[N Ward]

Thanks for this. Would this cause loss of data unless the VM was halted?

[Torsten. · Accepted solution]

The mirror/unmirror actions will prevent a data loss.

Question is, if all disks are internal, if you really need the c2... and c1... PVGs or if it is possible to mix. In this case you only need to adjust the lvmpvg file and mirror once.

[N Ward]

Fixed:

I edited the /etc/lvmpvg file to reflect the current situation

ie.

vmos_primary
c2t1d0
c2t2d0
c2t3d0
c1t4d0
vmos_mirror
c1t1d0
c1t2d0
c1t3d0
c2t4d0

I then mirrored all disks using vmos_mirror as the pv_path then reduced mirrors to 0 specifying c1t4d0 as the pv_path. I then put the /etc/lvmpvg file back to normal

vmos_primary
c2t1d0
c2t2d0
c2t3d0
c2t4d0
vmos_mirror
c1t1d0
c1t2d0
c1t3d0
c1t4d0

then remirrored using vmos_mirror as the pv_path. All fixed now..

[Torsten.]

Congratulations!


Next time avoid to reduce the disks from the VG, if possible.

[N Ward]

Indeed! pvchange is king :-) Thanks for your help..