Replacing a mirror boot disk (Real life scenario)

[Haitham Hamad]

I have a 2 related questions post. Last week we had to replace a mirror boot disk. I read most of the posts here regarding this issue and documented the following steps;

1. pvcreate -B /dev/dsk/rdsk/c2t10d0
2. vgcfgrestore -n /dev/vg00 /dev/dsk/c2t10d0
3. vgchange -a y /dev/vg00
4. mkboot /dev/rdsk/c2t10d0
5. mkboot -a "hpux -lq(;0)/stand/vmunix" /dev/rdsk/c2t10d0
6. lvlnboot -R
7. vgsync /dev/vg00

Was the sequence correct? ( I wasn't sure of doing 1 or 2 first).

The server I was working on is a remote server so I was working on the phone with HP's CE who was replacing the disk. My understanding is that the disk is hot-swappable so the CE just replaced the disk while the server is up and running. When I typed the vgcfgrestore command, I got all kind of errors. I then moved lvmtab to lvmtab.old and ran vgscan. The screen became garbled and it locked. After that, I was not able to access the server from my desktop and CE was not able to access it from the console. The server was still running normal and users can access the database. I then asked the CE to shutdown the server, put back the old disk and start on maintenance mode. I moved the lvmtab.old back to lvmtab and changed to level 3. Now everything is back to normal. I then broke the mirror, removed the disk from volume group, replaced the disk (server online), rebooted the server and build the mirror from the start. Everything is runnign fine now but need to an explanation of the 1 hour outage that happened.

Could someone explain what might have went wrong the first time. I thought the disk mightnot have been hot-swappable but I replaced it the second time while the server was up.

TIA,
Haitham

[Bill McNAMARA_1]

I tend to start all over again.

# vgreduce vg00 /dev/dsk/c1t5d0
Setup mirroring as normal:
# pvcreate -B /dev/rdsk/c1t5d0
# vgextend vg00 /dev/dsk/c1t5d0
# mkboot -l /dev/rdsk/c1t5d0
# mkboot -a "hpux (;0)/stand/vmunix" /dev/rdsk/c1t5d0
# for LVOL in /dev/vg00/lv*
> do
> echo $LVOL
> lvextend -m 1 $LVOL
> done

Later,

Bill

[Vincenzo Restuccia · Accepted solution]

1. pvcreate -B /dev/dsk/rdsk/c2t10d0
2. vgcfgrestore -n /dev/vg00 /dev/dsk/c2t10d0
3. vgchange -a y /dev/vg00
4. mkboot /dev/rdsk/c2t10d0
5. mkboot -a "hpux -lq(;0)/stand/vmunix" /dev/rdsk/c2t10d0
6. lvlnboot -R
7. vgsync /dev/vg00
Is correct in single user mode

[Patrick Wallek]

I have had 2 boot disks (mirrored) go bad in one system in the 2 months. The disks are in a Jamaica enclosure and I bring the system down prior to replacing the disk. I do this now because at one point I hot swapped a disk in a Jamaica and the system paiced, so I don't do that anymore.

I've done the replacement 2 different ways the 2 times I've done it:

1st time - I vgreduced the bad disk and then restarted from scratch on the mirrors.

2nd time - I did the steps that you talk about.
pvcreate -B
vgcfgrestore
mkboot commands
vgchange -a y /dev/vg00
vgsync /dev/vg00
lvlnboot -v to verify

and then I rebooted the system from the mirror to make sure it worked.

[James R. Ferguson]

Hi:

I've done this kind of replacement the way Bill suggested. I first 'lvreduce' (-m 0) to remove mirror copies, then 'vgreduce' the physical disk in question. At that point I know I have no I/O going to the disk I'm replacing. Then I physically replace the disk, and re-mirror.

I suspect that you didn't quiesce I/O to the disk before you replaced it, thus causing all the problems.

...JRF...

[Victor BERRIDGE]

Hi,
We did a hot swap disk change last week...

Here is the sequence form .sh_history:
ioscan -fnCdisk
vgdisplay -v /dev/vg00
ioscan -fn | more
ioscan -fnC disk
pvcreate -B /dev/rdsk/c0t8d0
vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t8d0
vgchange -a y /dev/vg00
vgdisplay -v vg00 | more
vgsync -v /dev/vg00

Then to be sure the disk was bootable:
mkboot /dev/dsk/c0t8d0
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c0t8d0
lvlnboot -Rv /dev/vg00
swapinfo
exit
Then yesterday evening I tested the reboot
It was working fine

All the best
Victor

[Haitham Hamad]

My main concern is not to reboot a production server to replace a mirror disk. I can always test the mirror on a scheduled downtime

From what I am hearing, correct me if I am wrong.

1) To use the vgcfgrestore, I have to be in single user mode. Reboot is not required

2) I can use the mirror rebuild procedure while in multi-user mode and Just swap the new disk. Reboot is not required

[Patrick Wallek]

Point #1 is not correct. You do NOT have to be in single-user mode to use vgcfgrestore. If the disk is truly hot swappable you should be able to replace the disk, do the vgcfgrestore and sync the mirrors with one being aware, maybe other than a slight performance hit, that it is going on.

Point #2 is correct, again assuming that the disk is truly hot-swappable and you don't panic the system like I did.

Just FYI - going to single user mode does not require a reboot, it can be done with an 'init s', but the system is inaccessable to users at that point since you have no networking daemons running. You are better off if you go to single user mode by rebooting, and interacting with the boot and doing an 'hpux -is'.

[Victor BERRIDGE]

Single user?
No I did this last week on a production system (I didnt have the choice) that is why it was rebooted later...

All the best
Victor

[Haitham Hamad]

Looks like either way should work on a production system without rebooting. I like James idea of ensuring there is no I/O coming into the disk before replacing it. Thanks everyone.

Haitham