topic Re: Boot disk failure/recovery in Operating System - HP-UX

Boot disk failure/recovery

Scott Guy_2 — Wed, 17 Jan 2007 14:38:27 GMT

Good afternoon gentlemen/ladies!!!

Ok, here's the picture. I've recently taken over admin services on an HP L2000. It appears we are having problems with the main boot disk, /dev/dsk/c1t2d0. Mirroring is being used, as I can see the mirror copies when doing a vgdisplay, etc. When I do an lvdisplay of all the different LVs in /dev/vg00, some of the extents on the main disk are stale, but all of the extents on the secondary disk are current.
So, here's my situation. I've read through notes in other posts on what needs to be done to recover (boot from alternate disk, etc.) but I have a bad feeling about just how the secondary disk is actually set up. I see it's mirrored, but I'm not 100% certain it's bootable.
I say this because when I run lvlnboot -v, I'm not getting anything back. Also, if I do a lifls on the alternate disk, it's only coming back with LABEL. To me, it looks like even though this disk is mirrored from an LVM perspective, I don't think I could boot from it.

I'm concerned. How can I be totally prepared when I go to replace the disk that seems to be bad??

Re: Boot disk failure/recovery

A. Clay Stephenson — Wed, 17 Jan 2007 14:57:12 GMT

Well, the way to be totally prepared is to test booting from an alternate disk BEFORE you actually need to. If lvlnboot -v /dev/vg00 does not display boot, root, and swap partitions on both the alterante and primary then it's a pretty good bet that they are not there.

I would approach your problem like this:

1) DO NOT SHUTDOWN! Slide your existing primary boot drive out a few centimters and allow it to spin down. Wait about 60 seconds. This will have the effect of converting your partially failed drive into a fully failed drive.

2) Remove the failed drive completely and insert the replacement drive.

3) vgcfgrestore -n /dev/vg00 /dev/rdsk/c1t2d0
4) vgchange -a y /dev/vg00
5) mkboot /dev/rdsk/c1t2d0
6) mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t2d0
7) lvlnboot -R
8) lvlnboot -v
and see if at least the primary boot data are displayed
9) vgsync /dev/vg00

Now, if lvlnboot -v still does not display the alternate boot data then I would unmirror all the LVOLS and then vgreduce the alternate disk out of vg00. Next run a "pvcreate -f -B" on the alternate disk and proceed to mirror the disk. I suspect that the "-B" pvcreate option was left out of the original mirroring procedure.

Re: Boot disk failure/recovery

DCE — Wed, 17 Jan 2007 15:04:46 GMT

First thing - make an ignite recovery tape. That way, if the mirror disk is not bootable, you will still be able to recover the system.

you can try the commands to copy the boot info and place the boot string in the mirror disk

mkboot â b /dev/rdsk/c2t0d0 /dev/rdsk/c1t0d0
copies the boot info from the original boot disk to the mirror disk (c2t0d0 in this case)

mkboot â a â hpux â lqâ /dev/rdsk/c1t0d0
places the boot string in the mirror disk

lvlvnboot to show the results

however, if the mirror disk was not created with the
pvcreate â B /dev/rdsk/c1t0d0
command, the disk is not bootable, nad the only way to make it bootable is to remove it from the vg and re-run the pvcreate command. Of course this will not work in your case, since the original boot disk is going bad

That leaves a couple of alternatives
1. If you another idle disk on the system, run the pvcreate and mkboot commands against it and creae a second mirrored disk.

2. make an ignite backup of vg00, replace the primary disk, and restore from the ignite tape (then mirror the disk properly

lvlnboot -vR /dev/vg00

Re: Boot disk failure/recovery

Scott Guy_2 — Wed, 17 Jan 2007 15:08:03 GMT

Thanks for the information. That's exactly what I needed.
When I saw that lvlnboot didn't show anything, I was afraid things weren't done quite right to begin with. I was afraid shutting down would put me in a bind.

Re: Boot disk failure/recovery

Scott Guy_2 — Tue, 23 Jan 2007 16:32:13 GMT

Clay,
I'm going to attempt this recovery tomorrow or Thurs.
Just to make sure I'm straight, once I replace the failed drive, my first command is the vgcfgrestore. I don't need to do a pvcreate first, correct??, since the vgcfgrestore should tell the disk it was the boot disk?
I certainly appreciate your help. Now I see why you're #1!!!

Re: Boot disk failure/recovery

A. Clay Stephenson — Tue, 23 Jan 2007 17:19:47 GMT

Yes, vgcfgrestore will restore the LVM metadata. You could execute a pvcreate -B before this operation but it would be pointless as the metadata would be overwritten by the vgcfgrestore command. If you want to feel extra safe you could do a make_tape_recovery or make_net_recovery first (but I'm sure that you don't need to do that because that is already one of your routine, automatic weekly or so tasks). In any event, if you follow the Yellow Brick Road I outlined above, the boot disk will be restored to its former self and no shutdown will be needed. In fact, I haven't shutdown a box to replace a failed disk in ~ 8 years and in that time, I've replaced tens of boot (or alternate boot) disks.

Re: Boot disk failure/recovery

A. Clay Stephenson — Tue, 23 Jan 2007 17:25:57 GMT

Oh, you also need to go out and buy yourself a baseball bat and give yourself two whacks -- hard. The first for running this long on a dead disk and the second for not having on-hand spare disks. Hang onto the bat because it is also a very useful tool for adjusting vendors, programmers, and DBA's.

Re: Boot disk failure/recovery

Scott Guy_2 — Wed, 24 Jan 2007 10:56:01 GMT

Yeah, the programmers are the reason I've waited this long!! They wanted to move some of their stuff to another server before I did anything. I told them the risks and they chose to roll the dice!!

Still, I probably deserve a good whack anyway.