topic Re: Replace HD in array - Preliminary steps ? in Operating System - HP-UX

Replace HD in array - Preliminary steps ?

TWBSupport — Mon, 24 Jul 2006 11:03:29 GMT

I have a couple bad sectors on a disk, I have mirrored the disk to another disk and now am ready to hot swap the bad disk with the new one. Are there any preliminary steps that I should do prior to just pulling the bad out and putting the new one in, besides have a backup on hand for the just in case. I'm assuming the mirror will kick in and syncronize the mirror to the new drive once placed into the slot the bad one came out of.

Should I do a fresh vgbackup or anything like that? Thank you in advance.

Re: Replace HD in array - Preliminary steps ?

James R. Ferguson — Mon, 24 Jul 2006 11:10:24 GMT

HI:

You don't provide any details -- LVM patches present, OS version, boot disk or not, so I'll point you to an excellent document:

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

You should already have a valid LVM configuration backup in '/etc/lvmconf' from your last LVM operation.

Regards!

...JRF...

Re: Replace HD in array - Preliminary steps ?

Jaime Bolanos Rojas. — Mon, 24 Jul 2006 11:11:22 GMT

Denda, are you using software mirror or hardware mirror?

What OS are you using and are you using LVM.

The logical volume that you mirrored is it in the vg00 or a data volume?

Regards,

Jaime.

Re: Replace HD in array - Preliminary steps ?

TWBSupport — Mon, 24 Jul 2006 11:18:25 GMT

Software mirroring w/MirrorDisk UX
yes, I'm using LVM
HPUX 11i
vg03 is the vg that contains the bad disk.

Re: Replace HD in array - Preliminary steps ?

Mancboy — Mon, 24 Jul 2006 11:40:06 GMT

First make your VG backups:
1) vgcfgbackup vgXX
2) vgexport -v -p -m vgXX.mapfile -f vgXX.discfile

IT'S IMPERATIVE THAT THE VGEXPORT USE THE -p FLAG - it means don't actually do anything nasty

Now remove the PV from LVM:
1) you should pvmove all LVs onto a new disc (You've already lvexended them, so just lvreduce -m 0 /dev/vgXX/LVNAME /dev/dsk/DUFF_DISC)

2) now that the disc is empty (pvdisplay /dev/dsk/DUFF_DISC should show "Allocated PE as being zero"

3) now vgreduce vgXX /dev/dsk/DUFF_DISC

Now take out the bad PV and put in a good one

1) pvcreate /dev/rdsk/NEW_DISC
2) vgextend /dev/vgXX /dev/dsk/NEW_DISC
3) for I in /dev/vgXX/lvol?
do
lvexend -m 1 $I /dev/dsk/NEW_DISC
done

OR you can just pull it out, put the new one in and lvsync each LV and HOPE ;-)

Re: Replace HD in array - Preliminary steps ?

Jaime Bolanos Rojas. — Mon, 24 Jul 2006 11:46:53 GMT

Denda, also this document in the itrc site details very good how to procced.

http://www1.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=LVMKBRC00006111

Regards,

Jaime.

Re: Replace HD in array - Preliminary steps ?

TWBSupport — Mon, 24 Jul 2006 12:42:10 GMT

OK - I'm reading over all the recommended material. I'll report back once I research a little more.

I just have one question and you all may laugh at me, but what good is a hot swappable disk if I have to have users log off the system that use that vg? Is 'hot swappable' only good for hardware mirroring?

Re: Replace HD in array - Preliminary steps ?

Jaime Bolanos Rojas. — Mon, 24 Jul 2006 13:13:43 GMT

Denda, you are not supposed to logoff regular users to do this procudure, since they are working on the volume group that was mirror on the good disk.

Hot swapping is an incredible benefit, since you can do so many things while you are online. The only difference between hardware mirror and software mirror in this instance, is that you have to play with commands a little to get the rudundancy up and going.
With hardware mirror you also gain performance.

Regards,

Jaime.

Re: Replace HD in array - Preliminary steps ?

TWBSupport — Mon, 24 Jul 2006 13:21:15 GMT

OK - I see what it's saying now. I apologize, I was skipping through and not completely reading everything.

"Replacing a bad mirrored disk by hot-swapping in a new disk can be a risky task, if the volume group is active and cannot be deactivated. If the disk is still attached to the volume group, the respective mirror copies of the affected logical volumes have to be reduced. "

I read the first line and then responded to the forum. As long as I break the mirror, I should be able to do all of this with the users online. Not a problem. I'll report back once I am done. Thank you.

Re: Replace HD in array - Preliminary steps ?

rick jones — Mon, 24 Jul 2006 19:39:54 GMT

It might be better to title the thread "Replace HD in enclosure/chassis" as I doubt what you have is an array. Were it an actual array rather than a hot-swap chassis all the LVM stuff would be moot (well, unless you were using the array in JBOD mode or something).

As you might have guessed from the thread, the difference between hot-swap and hot-plug is that hot-plug means simply that nothing will be fried electrically if you remove/replace the disc. There are still manual steps required before/after the replacement.

Hot-swap means it is all automagic - remove old disc, insert new disc. Happiness and joy, no explicit commands required.

In general, "enclosures" mean hot-plug and actual array controllers mean hot-swap.

Re: Replace HD in array - Preliminary steps ?

TWBSupport — Tue, 25 Jul 2006 11:47:40 GMT

Please do not shoot me. Here is what I am getting when I try to break the mirror on a couple of the lv's.
server1 # lvreduce -m 0 -A n /dev/vg03/lvol28 /dev/dsk/c4t10d0
Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vg03/lvol28".

lvdisplay shows the following
00461 /dev/dsk/c4t10d0 03260 current /dev/dsk/c4t9d0 02327 current
00462 /dev/dsk/c4t10d0 03261 current /dev/dsk/c4t9d0 02328 stale
00463 /dev/dsk/c4t10d0 03262 current /dev/dsk/c4t9d0 02329 current

Prior to me mirroring these to begin with a week ago, I knew c4t10d0 was the bad disk, since it was not mirrored at the time and we rec'd error's anytime we tried to read or write from a specific file system on the drive, I even deleted the lvol prior and recreated it and the same happened.

Now the error above looks as though c4t9d0 is the bad disk, or is it just because c4t9d0 can not read from c4t10d0 bad sectors, so it shows stale. If I have a stale lv on the disk that supposedly is good, how can I break that mirror?

Re: Replace HD in array - Preliminary steps ?

TWBSupport — Thu, 03 Aug 2006 11:03:30 GMT

just to give an update on how this was fixed just this morning.

Here were the steps:
mv lvmtab.XXX
vgscan
rm lvmtab
vgscan
vgcfgrestore -n /dev/vg03 /dev/rdsk/c4t9d0

vgchange -a y /dev/vg03

Bounced to single user mode
vgchange -a y /dev/vg03 which sync'd everything up (a little over 1 hour)
vg03 online and seeing all disks and mirrors properly