Re-mirroring root disk in an emergency onto another disk

[Philip Kime]

I am trying to automate the folowwing on 10.20:

I have a mirrored root disk, (c0t0d0 and c1t0d0). If c1t0d0 fails, I want to have a script perform the necessary actions to establish the mirror c1t7d0. After some testing, it's not clear to me how to handle the lvlnboot changes required because normally, root disk deaths are handled by replacing the disk with a new disk on the same place (same device file).

I had assumed (dead disk is PV key 1):

lvreduce -k 1 -m 0 /dev/vg00/lvolX
vgreduce /dev/vg00 /dev/dsk/c1t0d0
(or possibly vgreduce -f /dev/vg00 because the PV isn't queryable)
pvcreate -f -B /dev/rdsk/c1t7d0
mkboot -l /dev/rdsk/c1t7d0
mkboot -a "hpux -lq" /dev/rdsk/c1t7d0
vgextend /dev/vg00 /dev/dsk/c1t7d0
lvextend -m 1 /dev/vg00/lvolX /dev/dsk/c1t7d0

Remember, this is on a disk in a different place, not a replacement of the dead disk (because I might not be able to get on-site quickly). The first time I did this, the lvreduce of the first mirrored LV of the root VG (lvol1, /stand, boot area) hung.

So, the question is, how does lvlnboot interact with this? Do I need to lvrmboot -r, -b -s etc. first because I won't be re-mirroring with the same device?

The second test I did, I took out the dead disk from the root VG, pvcreated, mkbooted etc the new mirror, mirrored the boot LV and lvlnboot would not report the new mirrored boot LV, that is, I expected to see:

Boot: lvol1 on: /dev/dsk/c0t0d0
/dev/dsk/c1t7d0

but only saw

Boot: lvol1 on: /dev/dsk/c0t0d0

Any help appreciated.

[Claudio_17]

Hi,

have you tried with " lvlnboot -R " ?

[Philip Kime]

This completely hung the machine when I tried it! What exactly is lvlnboot -R supposed to do? I thought it simply restored the links when you replaced a dead mirror disk into the same slot? I'n not sure how it works when you put the new mirror in another slot (a different device file).

[Claudio_17]

I thought too .. sorry

What I know is that new boot disk is automaticaly recognized when added to root VG vg00.

What you've done is correct .. so I think you need lvrmboot -r, -b -s etc.

[Tim Adamson_1]

If you are worried about the time it will take to get to the server to fix the dead disk, why not have two-way mirroring in the first place?

The lvreduce would hang because you are missing a disk.

Anyway, have you taken the system to maintenance mode and forced the disk out of the vg. Then you should be able to pvcreate -B the new disk, vgextend, and then re-mirror all the lvols again.

Remember that when you reboot from maintenance mode, DO NOT init 3 as you may corrupt your good disk. Reboot the server.


Still not 100% sure of what you are trying to achieve, but maybe the suggestions will help. I know I have managed to reduce a missing mirror disk out of vg00 and I am sure I had to do it maintenance mode.


Tim

[Philip Kime]

Ah - I want to be able to do this without interrupting the machine at all. It's clearly possible for data disks but it looks like it might requre maintenance mode for boot disks?

All I'm trying to do is re-mirror a dead boot disk on an available spare without rebooting the machine or interrupting the service. Perhaps that's not possible though.

Two-way mirrors are tricky because of lack of disk slots ...

[Tim Adamson_1]

I don't think you will successfully be able to reduce out the dead disk without a reboot.

If you are short of disk slots, how were you going to add the other disk? And if it is on a SE SCSI, you can't use address 7.


Tim.

[Philip Kime]

These are rather special machines - Stratus DNCP hardware - there is no problem with slot 7 etc. Essentially, I'll have a spare disk already in there which I want to use to take over from any dead mirror disk, boot or not. It looks like perhaps it's only going to be possible with non-boot disks?

If so, this must be something to do with lvlnboot?

I mean, normally, you can lvreduce out the broken mirrored disk extents without a reboot -what would be the point of online mirroring if you couldn't?

[Bernhard Mueller]

Hi,

after lvextend -m 1 of all the lvols you should:
lvlnboot -b ${ROOTVG}/${BOOTLV}
lvlnboot -r ${ROOTVG}/${ROOTLV}
lvlnboot -s ${ROOTVG}/${SWAPLV}
lvlnboot -R

check with lvlnboot -v

Basically with the -k option the lvreduce works in most cases, but you cannot rely on that....

Additionally, you say you have some sort of extraordinary HW. Then you cannot be sure you can actually boot from your mirror until you tried it.

Finally, with 10.20 I had some bad experience a couple of years ago, because it would let you mirror the boot disk, boot from the mirror, let you take out the original boot disk from the Root VG, and then REFUSE to boot from that same (now unmirrored) disk.

(It just needed an LVM patch, though...).

Bottom line: you should have a reliable backup anyway.

Just my 0.02$

Regards,
Bernhard

[Philip Kime]

Answering my own question ...
We managed to do this with a script after all.
We run this from cron and it will detect dead mirrored disks, recommend where to re-mirror them until you van get on-site to replace the dead disk, and will create a script with the LVM commands to do the re-mirroring.

On 10.20, you have to do this for boot disks:

1: remove the mirrored LVs from the dead mirror using the PK key (using -A n too because vgcfgbackup fails since the dead disk is still in the VG it is trying to backup)
2: remove the disk from the VG
3: run lvlnboot -r, -b, -s and -d to reset these LVM links on the simplex disk
4: pvcreate the new root mirror
5: run mkboot -l and mkboot -a to populate the boot areas on the new root disk
6: vgextend the root VG with the new root disk
7: Mirror the root LVs onto the new root disk

This works well, completely on-line with no reboots. We were able to boot from either root disk after this, even though the new root disk was in a completely different slot to the original mirror. One thing to note is that lvlnboot -R is unecessary since any LVM command that changes the root LVs runs this automatically since HP-UX 10.0

Also, we noticed that lvlnboot -v does not show both root disks in the output for the Boot, Root and Swap areas until all three are mirrored. Strange but true.

[Steven E. Protter]

For this exercize to be of any value, you are going to have to run your script and during a maintenance window boot off this disk.

Just because lvlnboot -v says its a boot disk does not make it a boot disk. Booting off it makes it a boot disk.

Please consider assigning points to others that were actually helpful to you.

SEP

[Kent Ostby]

Keep in mind that if you hvae a disk that you want to mirror as a bootable disk, it has to have been created with the pvcreate -B option.

Best regards,

Kent M. Ostby

[A. Clay Stephenson]

You are really approaching this the wrong way. You are assuming that you can't replace a boot disk (or an alternate boot disk mirror) 'on the fly' BUT you can --- if all the disks in question are hot-pluggable. For example, on the old K boxes the internal disks were not hot-plug. Answer: dont use 'em but instead boot from external disks which are hot-pluggable. I have not shutdown in over 5 years to replace a boot disk (or its mirror) and over that period of time I have replaced tens of disks all without a single hiccup. The solution to your problem is to use hot-plug drives for everything and mirror. The probabiity of both drives failing before you can replace the defective drive is small but not zero. You could, of course, mirror twice OR (my preferred method) mirror once AND have a dd'ed 'lifeboat' copy of your boot disk ready to be plugged into the boot slot. The lifeboat protects you from two things that a mirror does not: 1) really, really bad patches. 2) Your own stupidity.

Food for thought, Clay

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.

[Philip Kime]

Just to cover some points above - I tested the results and an able to boot from either of the resulting disks running simplex (with -lq) or the resulting mirror so I'm happy. I think my scepticism was fuelled by some bad LVM patches on the machine (the patches needed reinstalling because swlist reported that they were there but the lvm commands were the wrong revision ...). This hardware is fully hot-swap so it all seems to work fine. Thanks for all replies.