topic Re: Replace root mirror disk,are these steps all right? in Operating System - HP-UX

Replace root mirror disk,are these steps all right?

hailerer — Tue, 20 Jan 2004 20:40:37 GMT

The engineer did following steps by online operation,after done we found many system file were damaged!If the steps are wrong,pls point out mistake so that they will be corrected.
1)first replace the bad disk online with hot pulling and pluging action.
2)ioscan -fnCdisk
3ï¼ mv /etc/lvmtab /etc/lvmtab.bak
4ï¼ pvcreate /dev/rdsk/c2t#d0
5ï¼ cp /etc/lvmtab.bak /etc/lvmtab
6ï¼ strings /etc/lvmtab
7ï¼ mkboot /dev/rdsk/c2t#d0
8ï¼ mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c2t10d0
9ï¼ vgcfgrestore -n /dev/vg00 /dev/rdsk/c2t10d0
10ï¼ vgsync /dev/vg00
11ï¼ lvlnboot -r /dev/vg00/lvol1
12ï¼

Re: Replace root mirror disk,are these steps all right?

hailerer — Tue, 20 Jan 2004 20:44:58 GMT

The engineer did following steps by online operation,after done we found many system file were damaged!If the steps are wrong,pls point out mistake so that they will be corrected.
1)first replace the bad disk online with hot pulling and pluging action.
2)ioscan -fnCdisk
3)mv /etc/lvmtab /etc/lvmtab.bak
4)pvcreate /dev/rdsk/c2t#d0
5)cp /etc/lvmtab.bak /etc/lvmtab
6)strings /etc/lvmtab
7)mkboot /dev/rdsk/c2t#d0
8)
mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c2t#d0
9)vgcfgrestore -n /dev/vg00 /dev/rdsk/c2t10d0
10)vgsync /dev/vg00
11)lvlnboot -r /dev/vg00/lvol1
12)lvlnboot -s /dev/vg00/lvol2

Re: Replace root mirror disk,are these steps all right?

Denver Osborn — Tue, 20 Jan 2004 21:29:31 GMT

Here's my 2 cents;

>1)first replace the bad disk online with >hot pulling and pluging action.

Before pulling out the "bad" disk, make sure it's dead and no IO could be going to it. Easy way is to test with dd, or diskinfo. On a completely dead disk, diskinfo would show the size as 0 bytes.

>2)ioscan -fnCdisk
>3)mv /etc/lvmtab /etc/lvmtab.bak
>4)pvcreate /dev/rdsk/c2t#d0
>5)cp /etc/lvmtab.bak /etc/lvmtab
>6)strings /etc/lvmtab

Not sure why you'd need to do all of this. after replacing it I'd probably take a look at ioscan to confirm we see the new disk. but moving the lvmtab and a pvcreate aren't making any sense.

>7)mkboot /dev/rdsk/c2t#d0
>8) mkboot -a "hpux -lq(;0) /stand/vmunix" /dev/rdsk/c2t#d0

Fine, if this was a bootable mirror for root vg.

>9)vgcfgrestore ->n /dev/vg00 /dev/rdsk/c2t10d0
>10)vgsync /dev/vg00
>11)lvlnboot -r /dev/vg00/lvol1
>12)lvlnboot -s /dev/vg00/lvol2

This is ok, after the disk is replaced you don't need to pvcreate it. Simply vgcfgrestore to put the lvm data back on the disk, then vgchange -a y, then you could check run vgsync if mirror shows stale extents.

The only thing that would worry me is pulling out a live disk that could possibly have IO. If the disk was not completely dead, you'd reduce the mirrors then replace disk.

There's also a good doc on the itrc re: lvm process for replacing failed lvm disk. I'm too lazy to look up the doc right now :) but search for "replace failed lvm mirror" and you should see come across it.

Hope this helps,
-denver

Re: Replace root mirror disk,are these steps all right?

Michael Tully — Tue, 20 Jan 2004 21:38:52 GMT

Here are the procedures from the technical knowledge base. The 'vgcfgrestore' does most of the work for you.

http://www2.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=KBAN00000347&search.y=3&search.x=21&mode=id&admit=-1335382922+1074652549821+28353475&searchCrit=allwords

Re: Replace root mirror disk,are these steps all right?

Steven E. Protter — Wed, 21 Jan 2004 00:46:13 GMT

I have a slightly more complete procedure for manually rebuilding the mirror.

You should not need it, but its good to have.

pvcreate -B /dev/rdsk/c1t0d0 #use real disk

mkboot -l /dev/rdsk/c1t0d0
mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c1t0d0 # use real disk

# mkboot -b /usr/sbin/diag/lif/updatediaglif -p ISL -p AUTO -p HPUX -p PAD -p LABEL /dev/rdsk/c?t?d?

If you are running 64-bit OS:

# mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p AUTO -p HPUX -p PAD -p LABEL /dev/rdsk/c?t?d?

vgextend /dev/vg00 /dev/dsk/c1t0d0 # same thing
lvextend -m 1 /dev/vg00/lvol1 /dev/dsk/c1t0d0

# real disk. repeat for other lvols

lvlnboot -r /dev/vg00/lvol3 # root fs /
lvlnboot -s /dev/vg00/lvol2 #swap
lvlnboot -d /dev/vg00/lvol2 #swap/dump
lvlnboot -b /dev/vg00/lvol1
lvlnboot -R
lvlnboot -v
setboot
setboot -a 52.1.0 # second disk

Thanks to all that made this doc right.

SEP

Re: Replace root mirror disk,are these steps all right?

Phil Smith — Wed, 21 Jan 2004 19:28:00 GMT

Something I think a lot of people overlook is that in the event of lossing one of your mirrored OS disks it you still want the server to be able to boot unattended from the remaining disk you need to specify -lq (quorum override) as with one disk gone you don't have a quorum.

Note this should be set for BOTH the primary and secondary disk.

Re: Replace root mirror disk,are these steps all right?

Sridhar Bhaskarla — Thu, 22 Jan 2004 00:01:48 GMT

Hi,

YOu got the mirroring steps. Make sure you follow those links/steps to mirror your disks correctly. Do not reboot your system until "lvlnboot" is fixed correctly.

lvlnboot -r /dev/vg00/lvol1 is a blunder unless you really made your root logical volume as lvol1. Usually it is stand and he should be giving 'lvlnboot -b /dev/vg00/lvol1', 'lvlnboot -s /dev/vg00/lvol2', 'lvlnboot -r /dev/vg00/lvol3' and 'lvlnboot -d /dev/vg00/lvol2'.

You didn't have to move lvmtab file.

-Sri

Re: Replace root mirror disk,are these steps all right?

Kyri Pilavakis — Thu, 22 Jan 2004 02:17:38 GMT

I have been looking after an HP-UX systems and I have a document which I have set up. This has been very useful to me when I had various disk failures.

Sending this to you and I hope it will also be of some use. It contains other stuff which might also be of use. Unfortunately not to me any more as our company scraped all unix systems and moved to NT and VMS !!

Please dont take it as bible...I am only human !!

kyris

Re: Replace root mirror disk,are these steps all right?

James Lynch — Thu, 22 Jan 2004 09:17:22 GMT

I agree with Denver, you should never hot-plug a disk that is not completely dead. If there is any possibility that the suspected failed disk can handle an I/O request, then Murphy will strike and you will end up with filesystem corruption.

Hot-plug does not necessarily equate with hot-swap. Hot-plug means that the hardware device can support removal and insertion of the device with out having to remove power from that device, in this case a disk and it's enclosure. Hot-swap refers more to the OS's ability to support device removal and insertion of a device while I/Os are still active. HP's LVM does not support this hot swap capability and was never designed with this capability in mind.

If you are going to attempt to hot-plug a disk, then you must make sure that all I/Os to the lvols on that disk have stopped. There are three ways to do that, none of which were the method your engineer used:

1) Shutdown the system and relpace the disk.
or
2) Unmount all filesystems that are contained in the lvols on the affected disk. Also stop any application that is using the lvols in RAW mode, i.e. Database.
or
3) lvreduce the failed disk's mirrors and then vgreduce the failed disk from the VG. Depending upon the version of the OS and how the disk has failed, this may or may not be possible.

I have successfully used all three of the mthods above. I also have tried testing replacing a disk with the method that you used, and I was able to introduce corruption in my mirrored filesystems.

Thinking that you can replace LVM mirrored disk while the filesystems are mounted and the failed disk is still active in the volume group seems to be a common misperception. I have run into several people have tried to do this and ended up with corrpted filesystems.

JL

Re: Replace root mirror disk,are these steps all right?

A. Clay Stephenson — Thu, 22 Jan 2004 10:30:19 GMT

I think the fundamental problem was that no vgchange was done. The pvcreate was unnecessary. I have replaced at least a hundred "flakey" boot drives without a problem by yanking them out with a measure of caution. Those who advocate shutting down to replace hot-plug drives should consider this. When is a drive most likely to fail? I know of at least two occasions when the remaining drive failed to become active after a shutdown --- leaving a dead box. I also fail to see how a box can distinguish between a yanked-out drive and a truly dead one. I trust tri-state electronics and have no fear of pulling a drive whether completely dead or not BUT one must take a few precautions. It is possible to be in a state where stale extents are present on both mirrors --- that is where the corruption ultimately comes from.

1) Do a series of lvdisplay -v commands for each lvol on the physical disk and make sure that all the extents on the remaining "good" drive are current. (I actually have a script for this --- checkextents.sh)

2) "Yank" the bad or flakey drive out. Pull it out just 2 cm or so and leave it resting in the slot. This lets the drive spin down gradually and who knows you might need it again.

3) Wait 30 seconds or so and then run checkextents.sh agian making sure that the remaining drive has all "current" extents.

4) Now remove the bad drive completely from the slot and insert the replacement drive. Allow it to spin up. Think happy thoughts.

Let's assume the replacement drive is c0t6d0:
5) vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t6d0
6) vgchange -a y /dev/vg00
7) mkboot /dev/rdsk/c0t6d0
8) mkboot -a "hpux -lq (;0)/stand/vmunix" /dev/rdsk/c0t6d0
9) lvlnboot -R
10) vgsync /dev/vg00 --- this could take a few tens on minutes

11) Do lvdisplays to make certain that all extents are current.

I have yet to have the above procedure fail but maybe I've just been lucky.

Re: Replace root mirror disk,are these steps all right?

James Lynch — Thu, 22 Jan 2004 11:56:22 GMT

Clay,

I don't doubt that you have successfully used your procedure for replacing many faulty disks. I would also say that you have been very lucky up to this point to never have seen any corruption.

The problem that I refer to has nothing to do with the tri-state electronics ability to electrically isolate the disk being replaced, but has everything to do with how LVM handles the removal and insertions of disk devices.

Depending upon the current state of the failed/flakey disk when it is removed, LVM may or may not be able to differentiate between the disk that was just removed and the new one that was just inserted. The reason is that LVM, up to this point, has not been told that one of it's disks has just been replaced. The failed/flakey disk could have been in a power failed state, when the new replacement disk is inserted, LVM see its disk at that same H/W path come back from a powerfail. Any I/O that was pending for that disk will now be sent. But wait, this new disk has not yet been preped for use by LVM, that will happen as soon as you run the vgcfgrestore command. So now you have LVM thinking that the pending I/Os were successfully written to the disk, so it in turn updates the extent maps to indicate that the extent is no longer stale. LVM thinks that the extent is now in synch between both disks. This isn't so much of a problem because vgfcgrestore will mark all of the extents as stale on the new replacement disk and they will get resynched by vgsync or lvsync.

You can clearly see that from the time you insert your new disk to the time that you run vgcfgrestore, there is a small period of time where LVM thinks that the new disk is a valid disk to use. The real problem comes into play with any I/Os that read from the affected lvol during this window of vulnerabilty. Because LVM only marks an extent as stale when it can not write it to a disk, there is a high probability that the failed disk's extent map shows the majority of extents as current and not stale. There will probably be only a handful of extents marked stale. Along comes a read I/O. LVM looks at the extent map that is stored in kernel memory to determine where to retrieve the I/O from. Remember that read I/Os are not sent to both mirrors, but are only sent to the least busy mirror. There is a 50% chance that LVM will send the I/O request to the uninitialized replacement disk. LVM gets its I/O request satisfied by reading garbage from the new disk. This garbage data block is used by the filesystem, or application, processed and then most likely written back out to the lvol. Now that you have read garbage in, that same garbage now gets propagated out to both of your mirrors. LVM is happy in that it thinks that the data (LVM doe not care that the data is garbage) was successfully written to both mirrors. Now you have corrupted data on both disks.

The procedures that I described are consistent with HP's documented procedures for replacing a failed mirrored LVM disk.

I hope that this helps to clear up the reason as to why there must not be any active I/Os on the lvols.

JL