Replacing a Failed Primary LVM Mirrored bootdisk on Hot-Swappable Enclosure

[Alzhy]

I know this is very basic to some of you. I searched ITRC and the recovery handbook but ended up confused as to the "proper way" of replacing a failed OS mirror (a primary at that but I guess it really does not matter).

HP already replaced the failed drive and we can verify it is there. However we seem to be at a loss on how to proceed further in the light of the many divergent "recipes" out there. "vgdisplay v vg00" shows the disk as unavailable and the mirrors as stale. I can still do pvdisplay on the failed disk. I am using the following process to complete the LVM steps:

1. pvcreate -B /dev/rdsk/cXtYdZ # N is [RL] no.
2. mkboot -l /dev/rdsk/cXtYdZ
3. mkboot -a "hpux - lq ;0)/stand/vmunix" /dev/rdsk/cXtYdZ
4. vgcfgrestore -n vg00 /dev/rdsk/cXtYdZ
5. vgsync vg00

However at step 1, it fails with:

# pvcreate -B /dev/rdsk/c30t2d0
pvcreate: The physical volume "/dev/dsk/c30t2d0" is already recorded in the "/etc/lvmtab" file.


Also, there are recipes out there (specifically the handbook) wherein it says that if the failed PV is still attached, the process would be to lvreduce the LVOLS and then vgreduce the failed PV and simply redo the mirroring.

So which is which?

[Uday_S_Ankolekar]

I would use lvreduce on all lvols first and then vgreduce on the perticular new disk. And after do all necessary things to add new disk in vg for creating a mirrored boot disk

pvctreate -f B /dev/rdsk/devname, followed by mkboot, vgextend , lvextend -m on all lvols, lvlnboot.
I am not giving you entire command assuming you know it already..

[RAC_1]

I also use,
mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p HPUX -p AUTO -p LABEL -p PAD
/dev/rdsk/cxtxdx
This takes care of installing ODE tools on the disk.

Anil

[RAC_1]

Before you do pvcreate -B /dev/rdsk/c30t2d0
do vgreduce -f vg00
then check lvmtab (strings /etc/lvmtab) and then proceed to pvcreate -B /dev/rdsk/c30t2d0

[Patrick Wallek]

I generally do:

1) vgcfgrestore
2) mkboot -a
3) mkboot -b
4) vgsync
5) lvlnboot -v
to verify that everything is still OK.

Then at the first available opportunity boot the machine and boot from the disk that was replaced just to verify that it still works.

[Alzhy]

Patrick, et al.

The HP Software Recovery Handbook even has a flowchart of sorts to check if a PV is attached or non-attached.

Normally, when a disk fails (lbolst.. etc.), and the disk is hot-swappable, after the disk is physiclaly replaced (ioscans done, etc...) - what state is that failed disk at usually? - as described in the "handbook"? Inmy case it appears that it is "attached" so I would have to follow the recipe of lvreducing the mirrors, taking the PV out of the VG00 and redoing the mirrors from scratch.

Why then the various recipes? Those that even suggest a pvreate -B.. to a vgsync? I know I have used such method in the past I am still stumped on this one - and I need to write up a procedure for our team on how to handle bootdisk failures on a hot-swap environment.

[Uday_S_Ankolekar]

Usually you don't need to lvreduce or vgreduce but without reducing lvol some time vgsync errors out ( It happend to me )

[Geoff Wild]

This is another "that depends" question:

After failed disk has been replaced by HP (assuming the disk is c1t5d0):

vgcfgrestore -n /dev/vg00 /dev/rdsk/c1t5d0

mkboot /dev/rdsk/c1t5d0

mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c1t5d0

vgchange -a y /dev/vg00

lvlnboot -Rv /dev/vg00

vgsync /dev/vg00


1) If the disk is completely "dead", such as if you run ioscan and status is "no_hw" then you can hot swap the disk online.

2) However in circumstances where the disk has not fully failed please do one of the following to avoid data corruption :

a) reduce mirror before replacing the disk
b) deactivate VG before replacing the disk
c) shutdown system to replace the disk


Rgds...Geoff

[Florian Heigl (new acc)]

Nelson,

I'm also not very happy about the various procedures, but there some reasons I can think of:

the lvreduce / vgreduce way is the cleanest which hp technicians will always follow.

It ensures that no problems due to stale PEs can go unnoticed. (imagine a PE that never got properly mirrored resided on the failed drive), and HP needs their technicians to follow a step-by-step procedure that keeps them safe from any liability issues :)

many people that are doing this on a more regular basis will simply go via using vgcfgrestore and vgsync as they know/trust they'll notice a stale PE at a glance.
the lvextend -m 1 after reducing first is an unnecessary risk in their eyes, as the unmirrored time will increase a bit.

If You are to write a full-blown procedure then You'll be best of to go through a lot of testing and chose the way that fits Your environment.

maybe the best thing would rely on an analisys of the error state.
i.e. a disk with a failing motor will be available and drop out again, in that case the top priority IS breaking the mirror and pulling it out.
if You've only seen a few bad blocks, You won't care that much and will go for the more consistent way of lvreducing it before the hp tech arrives.

[Alzhy]

Florian,

Thanks for your thoughts and we will now henceforth make it standard practice on our environments (since all are hot-swappable) to simply follow the safe approach and that is to redo the mirror after reducing the VG by the replaced disk..

I believe in our case wherein an HP CE would usually just come and replace a disk physically without even doing any LVM gyrations - that this is the safe approach.

[A. Clay Stephenson]

Essentially I do the steps as outlined by Geoff with one exception. I run a script which makes sure that all the extents in every LVOL in vg00 are "current" in the remaining good disk. In the case of a failing drive (as opposed to a failed drive), I convert ALL failing drives to failed by the expedient of pulling the flaky drive out a few centimeters and letting it spin down. Wait a few tens of seconds and now you have a failed drive.

You have now reduced your boot disk replacement procedure to one routine that doesn't branch.

I always keep at least one spare of every drive model in use (even though I am under full support) so that I don't have to wait from the HP Mr. Goodwrench to show up. They simply hand me (or ship me) a drive that goes back into spare parts.

Fundamentally, your original procedure should never have included the pvcreate as the vgcfgrestore takes care of that for you.

[Kyris]

Here are the steps I take to replace a mirrored boot disk - not hot swappable but so ignore the first 2 step.

1 â replace with a new one

2â Rebooted machine to multi user mode. Failed to reboot so went to single user mode â interrupted boot process ï   ISL > hpux â lq ( quorum problem). Booted okâ ¦

3 vgcfgrestore -n /dev/vg03 /dev/rdsk/c0t4d0 : this restores LVM configuration info on the disk. Always gives an error BUT then says itâ s done- ignore error!!!

4 mkboot -l /dev/rdsk/c0t4d0 : places boot utilities in boot area of the disk

5 mkboot -a â hpux â lq (;0)/stand/vmunixâ /dev/rdsk/c0t4d0 : points disk to /stand dir

6 lvlnboot â R : recovers missing links to all of logical volumes

7 vgchange -a y /dev/vg03 : to reactivate the volume group to which the disk belongs

8 vgdisplay â v /dev/vg03 | more

9 vgsync /dev/vg03 [&] : to manually sychronize all extents in the volume group.

10 lvdisplay â v /dev/vg03/lvoln : to see lvols as they are synced are done ok

11 Check with lvlnboot -v : to see both disks

12 Check with setboot to see itâ s secondary ..

13 Shutdown â h â y 0 : t

[Rory R Hammond]

Since 11i
If I install a disk that had old information,
I have to do /sbin/pvremove /dev/rdsk/xxxxx
Then the "other stuff works"

Rory