LVM and VxVM

Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

 
Alzhy
Honored Contributor

Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

I know this is very basic to some of you. I searched ITRC and the recovery handbook but ended up confused as to the "proper way" of replacing a failed OS mirror (a primary at that but I guess it really does not matter).

HP already replaced the failed drive and we can verify it is there. However we seem to be at a loss on how to proceed further in the light of the many divergent "recipes" out there. "vgdisplay v vg00" shows the disk as unavailable and the mirrors as stale. I can still do pvdisplay on the failed disk. I am using the following process to complete the LVM steps:

1. pvcreate -B /dev/rdsk/cXtYdZ # N is [RL] no.
2. mkboot -l /dev/rdsk/cXtYdZ
3. mkboot -a "hpux - lq ;0)/stand/vmunix" /dev/rdsk/cXtYdZ
4. vgcfgrestore -n vg00 /dev/rdsk/cXtYdZ
5. vgsync vg00

However at step 1, it fails with:

# pvcreate -B /dev/rdsk/c30t2d0
pvcreate: The physical volume "/dev/dsk/c30t2d0" is already recorded in the "/etc/lvmtab" file.


Also, there are recipes out there (specifically the handbook) wherein it says that if the failed PV is still attached, the process would be to lvreduce the LVOLS and then vgreduce the failed PV and simply redo the mirroring.

So which is which?

Hakuna Matata.
12 REPLIES 12
Uday_S_Ankolekar
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

I would use lvreduce on all lvols first and then vgreduce on the perticular new disk. And after do all necessary things to add new disk in vg for creating a mirrored boot disk

pvctreate -f B /dev/rdsk/devname, followed by mkboot, vgextend , lvextend -m on all lvols, lvlnboot.
I am not giving you entire command assuming you know it already..
Good Luck..
RAC_1
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

I also use,
mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p HPUX -p AUTO -p LABEL -p PAD
/dev/rdsk/cxtxdx
This takes care of installing ODE tools on the disk.

Anil
There is no substitute to HARDWORK
RAC_1
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

Before you do pvcreate -B /dev/rdsk/c30t2d0
do vgreduce -f vg00
then check lvmtab (strings /etc/lvmtab) and then proceed to pvcreate -B /dev/rdsk/c30t2d0
There is no substitute to HARDWORK
Patrick Wallek
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

I generally do:

1) vgcfgrestore
2) mkboot -a
3) mkboot -b
4) vgsync
5) lvlnboot -v
to verify that everything is still OK.

Then at the first available opportunity boot the machine and boot from the disk that was replaced just to verify that it still works.
Alzhy
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

Patrick, et al.

The HP Software Recovery Handbook even has a flowchart of sorts to check if a PV is attached or non-attached.

Normally, when a disk fails (lbolst.. etc.), and the disk is hot-swappable, after the disk is physiclaly replaced (ioscans done, etc...) - what state is that failed disk at usually? - as described in the "handbook"? Inmy case it appears that it is "attached" so I would have to follow the recipe of lvreducing the mirrors, taking the PV out of the VG00 and redoing the mirrors from scratch.

Why then the various recipes? Those that even suggest a pvreate -B.. to a vgsync? I know I have used such method in the past I am still stumped on this one - and I need to write up a procedure for our team on how to handle bootdisk failures on a hot-swap environment.

Hakuna Matata.
Uday_S_Ankolekar
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

Usually you don't need to lvreduce or vgreduce but without reducing lvol some time vgsync errors out ( It happend to me )
Good Luck..
Geoff Wild
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

This is another "that depends" question:

After failed disk has been replaced by HP (assuming the disk is c1t5d0):

vgcfgrestore -n /dev/vg00 /dev/rdsk/c1t5d0

mkboot /dev/rdsk/c1t5d0

mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c1t5d0

vgchange -a y /dev/vg00

lvlnboot -Rv /dev/vg00

vgsync /dev/vg00


1) If the disk is completely "dead", such as if you run ioscan and status is "no_hw" then you can hot swap the disk online.

2) However in circumstances where the disk has not fully failed please do one of the following to avoid data corruption :

a) reduce mirror before replacing the disk
b) deactivate VG before replacing the disk
c) shutdown system to replace the disk


Rgds...Geoff
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
Florian Heigl (new acc)
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

Nelson,

I'm also not very happy about the various procedures, but there some reasons I can think of:

the lvreduce / vgreduce way is the cleanest which hp technicians will always follow.

It ensures that no problems due to stale PEs can go unnoticed. (imagine a PE that never got properly mirrored resided on the failed drive), and HP needs their technicians to follow a step-by-step procedure that keeps them safe from any liability issues :)

many people that are doing this on a more regular basis will simply go via using vgcfgrestore and vgsync as they know/trust they'll notice a stale PE at a glance.
the lvextend -m 1 after reducing first is an unnecessary risk in their eyes, as the unmirrored time will increase a bit.

If You are to write a full-blown procedure then You'll be best of to go through a lot of testing and chose the way that fits Your environment.

maybe the best thing would rely on an analisys of the error state.
i.e. a disk with a failing motor will be available and drop out again, in that case the top priority IS breaking the mirror and pulling it out.
if You've only seen a few bad blocks, You won't care that much and will go for the more consistent way of lvreducing it before the hp tech arrives.
yesterday I stood at the edge. Today I'm one step ahead.
Alzhy
Honored Contributor

Re: Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

Florian,

Thanks for your thoughts and we will now henceforth make it standard practice on our environments (since all are hot-swappable) to simply follow the safe approach and that is to redo the mirror after reducing the VG by the replaced disk..

I believe in our case wherein an HP CE would usually just come and replace a disk physically without even doing any LVM gyrations - that this is the safe approach.

Hakuna Matata.