topic Re: Replacing Internal Hot-Swap Drive on an rp7400 in Operating System - HP-UX

Replacing Internal Hot-Swap Drive on an rp7400

Andrew Griffin — Tue, 10 Jun 2003 20:07:22 GMT

Ok... Yesterday morning the primary internal disk on one of our rp7400's died. I rebooted off of the second drive, which was mirroring the primary drive, and everything came back up. I have since recieved the replacement drive, and swapped out the failed drive with it. While I've been an HP-UX admin for about 4 years now, this is the first time I've done this, and I want to make sure I don't screw anything up seeing as this is the PeopleSoft server (which cuts our paychecks :)) All I've done is a:

pvcreate -B /dev/rdsk/c1t6d0

And I'm not quite positive where to go from here. The backup disk (which it booted off of) is:

/dev/dsk/c2t6d0

What do I need to do to get the new disk mirrored and synced up?

Re: Replacing Internal Hot-Swap Drive on an rp7400

Sridhar Bhaskarla — Tue, 10 Jun 2003 20:17:44 GMT

Hi,

It's quite easy and straightforward with hot-swappable disks. Below is the procedure.

#vgcfgrestore -n vg00 /dev/rdsk/c1t6d0
#vgchange -a y vg00
#mkboot -l /dev/rdsk/c1t6d0
#cd /usr/sbin/diag/lif
(update offline diag tools)
#mkboot -vb updatediaglif2 -p ISL -p AUTO -p HPUX -p LABEL /dev/rdsk/c1t6d0
#mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t6d0
#lvlnboot -b /dev/vg00/lvol1
#lvlnboot -s /dev/vg00/lvol2
#lvlnboot -r /dev/vg00/lvol3
#lvlnboot -d /dev/vg00/lvol2
#lvlnboot -R
#lvlnboot -v

(make sure both the disks are showing up as boot disks)

Now sync up the mirrors

#vgsync vg00
#vgchange -a y vg00

Verify that there are no stale entries

#vgdisplay -v vg00 |grep stale

-Sri

Re: Replacing Internal Hot-Swap Drive on an rp7400

Patrick Wallek — Tue, 10 Jun 2003 20:20:26 GMT

Here's what I do:

# pvcreate -f -B /dev/rdsk/c1t6d0
# vgcfgrestre -n vg00 /dev/rdsk/c1t6d0
# mkboot /dev/rdsk/c1t6d0
# mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t6d0
# vgchange -a y vg00
# vgsync

# lvlnboot -v

just to verify that everything is still OK after the above steps.

All this can safely be done while the machine is up and people are working.

Also just a point -- If your disks were mirrored correctly, you should not have had to reboot in order for the mirror to take over. It should have just taken over automagically.

Re: Replacing Internal Hot-Swap Drive on an rp7400

A. Clay Stephenson — Tue, 10 Jun 2003 20:20:58 GMT

The pvcreate was not necessary.

1) vgcfgrestore -n /dev/vg00 /dev/rdsk/c1t6d0
2) vgchange -a y /dev/vg00
3) mkboot /dev/rdsk/c1t6d0
4) mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t6d0
5) lvlnboot -R
6) vgsync /dev/vg00

When possible, shutdown and make certain that you can boot from the new disk.

I'm a little surprised that you needed to reboot because the box should have stayed up.

P.S. Truly paranoid Sysadmins will keep spare drives on-hand even when under maintenance.

Re: Replacing Internal Hot-Swap Drive on an rp7400

Gino Casella — Tue, 10 Jun 2003 20:28:38 GMT

Hi,
there are two ways of getting c1t6d0 return the primary boot disk.

The first one is proceding as mirroring it from the secondary disk:
first extend the vg (vgextend /dev/vg00 /dev/dsk/c1t6d0)
then make HP_UX know that disk is bootable and where finding the kernel (mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t6d0);
then extend all logical volumes

- lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c1t6d0
- lvextend -m1 lvolx
- lvext...
..
- lvextend -m 1 lvol9 on c1t6d0

the last thing is make the three firs lvol being the boot ones (lvlnboot -Rv /dev/vg00)

The second way is backing up the vg structure from a previously backed up file (vgcfgrestore)

Gino

Re: Replacing Internal Hot-Swap Drive on an rp7400

Andrew Griffin — Tue, 10 Jun 2003 23:09:10 GMT

All good suggestions - I called in a software support call before I had a chance to re-check this thread and the software support person said basically the same things. However, I needed to boot in to single-user mode to do it. Now I have an even bigger problem, I can't get the system to boot in to single-user mode. After I put in "hpux -is -lq" in the ISL menu, it panics on a VFS_MOUNTROOTFSs type of error. I've seen this before - you know how when you do a "lvlnboot -v" command, and it tells you the boot, root, and swap logical volumes. I've seen the exact same errors on a system where the boot logical volume information was dropped because of a mis-type lvlnboot command. Now I'm trying to boot from the install CD and I'm going to try to recover this lvlnboot setting via a recovery session. However, it's taking forever for it to boot from the CD, it's been booting for ~15 minutes now.... Please, any suggestions are more than welcome.

Re: Replacing Internal Hot-Swap Drive on an rp7400

Andrew Griffin — Tue, 10 Jun 2003 23:11:30 GMT

The first time I saw this error, I wasn't the one recovering the system and the other sys admin ended up having to re-install the system. It was only after he was done re-installing the system that he figured out that the culprit was the lvlnboot command. What should I do once I get this booted in to the recovery shell?

Re: Replacing Internal Hot-Swap Drive on an rp7400

Andrew Griffin — Tue, 10 Jun 2003 23:13:52 GMT

or maybe there's a way to specify the boot logical volume in the boot menu or ISL menu? It's still booting off of the CD - up to ~30 minutes now... It's stopped at the memory information section - there's 10Gb of memory in this system, could that be what's causing it to take so long?

Re: Replacing Internal Hot-Swap Drive on an rp7400

Patrick Wallek — Tue, 10 Jun 2003 23:29:50 GMT

Don't worry about being in single-user mode. There really is no real need for it. Go ahead and boot back up into multi-user mode and go ahead with the procedure.

This is a "hot-swap" drive so really there is never any need to reboot at all if you are correctly mirrored.