topic Re: what is the best way of replacing disk that is "kind of" broken in Operating System - HP-UX

what is the best way of replacing disk that is "kind of" broken

Kirk Reindl — Wed, 31 Jul 2002 17:47:12 GMT

L2000
HPUX OS 11.11
Disk size: 36GB
Part of HP 2100 disk rack.
Disk's config in LVM: extent based striping with mirroring.
Number of logical vols the disk belongs to: 30

I have a disk in a volume group that shows a read error in stm and is reporting a SCSI read error in the syslog. I would like to have the disk physically replaced and I'm looking for the best, most efficient way to get the old disk out the new disk configured. I can still talk to the disk, for example a vgdisplay -v does not detect a problem. That is why I say the disk is ???kind of??? bad. Nonetheless, I need to get the disk out.

In the past, I have merely pulled the bad disk out, put a new disk in, and did vgcfgrestore, vgchange, vgsync, etc. However, I have gotten burned on this in the past as I had an instance where LVM did not mark all the extents on the bad disk as being stale after I had pulled it. Well my app went to write to disk thinking the extents were current, which they were not (the disk was pulled), and my app blew up.

The only other technique I'm aware of is lvreducing the bad disk out, then vgreduce/extend, then lvextend. Well as you can imagine, doing this across 30 lvols is quite cumbersome.

I guess my question is simply, is there a better, more efficient way to get this disk out and get the new one in??

Any ideas or direction is appreciated.

Thanks
Kirk Reindl

Re: what is the best way of replacing disk that is "kind of" broken

Steven Mertens — Wed, 31 Jul 2002 17:52:18 GMT

hi,

maybe you can first use the pvmove
command, and move alle used extents to a disk
or disks where space is available.
If all extents are free on the broken disk,
you can remove him from the vg.

regards,

Steven

Re: what is the best way of replacing disk that is "kind of" broken

A. Clay Stephenson — Wed, 31 Jul 2002 17:55:47 GMT

Hi Kirk:

I have so many hot-plug drives that I generally feel something is wrong if I don't replace at least one every two or three weeks. I never do the lvreduce. I start with the vgcfgrestore and have never had the kind of problem that you describe. In your case, where the disk has not failed, I suggest that you pull the disk and leave it out for a miute or two before replacing the disk with the new one. That way, LVM and the IO subsystem definitely knows there is a problem.

Of course, the safe advice is always have a known good backup before ... . In reality, I replace these frequently and never give it a thought.

Regards, Clay

Re: what is the best way of replacing disk that is "kind of" broken

S.K. Chan — Wed, 31 Jul 2002 17:58:52 GMT

If it's "kindda broken" then you have a least some time to plan for this. The safest/cleanest way is to do this in single-user mode. That way you can be sure no one is accessing the disk. As far as the process goes..

After the disk is physically replaced boot up in single-user mode and run .. (assuming vg02 and the disk is c2t2d0)

# vgcfgrestore -n vg02 /dev/rdsk/c2t2d0
# vgchange -a y vg02
# vgsync vg02

That's about the simplest it could get. The usual applies, make sure full backup is done before doing this and most importantly make sure your system is patched with the latest LVM related patches.

Re: what is the best way of replacing disk that is "kind of" broken

Sajid_1 — Wed, 31 Jul 2002 18:02:42 GMT

hi,

I would think the best plan would be:

take good full backup
find out the LVs in the *bad* disks, create a VG map of the VG with vgexport -p:
# vgexport -p -v -s -m mapfile
This map file can be used incase of any issues while restoring VG
Remove disk from the system
Add new one and do a vgcfgrestore. Check the disk with lvdisplay -v and find out any stale on that. If found do a vgsync or lvsync.

rgds,

Re: what is the best way of replacing disk that is "kind of" broken

Sajid_1 — Wed, 31 Jul 2002 18:20:51 GMT

hi,

This document describes the disk replacement procedures in detail (TKB #KBRC00009115):

http://support1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000059338243

rgds,

Re: what is the best way of replacing disk that is "kind of" broken

Kirk Reindl — Wed, 31 Jul 2002 18:31:14 GMT

PLEASE: All repliers to this question, read the following

sid:

Thanks for the link to the document. I know I had an instance where I corrupted a database by trying to pull the broken disk, and your doc mentions this. I wish I knew why this happened, but at least I know I'm not going crazy, not yet anyway.

Regards,

Kirk

Re: what is the best way of replacing disk that is "kind of" broken

Chris Gromala — Thu, 10 Apr 2008 15:32:05 GMT

I know this is an old problem but I found the fix after a long time...

If a disk is reporting errors and is still up you have to perform a pvchange on the disk device before physically taking it out. This is the list of commands I used after struggling with this for a while. I hope this works for anyone in the same boat...

Considering the disk is /dev/dsk/c3t5d0

Here is the order of cmds actions (slight change of when the vgcfgrestore and pvchange is done):

1. Deactivate the PV:
# pvchange -a N /dev/dsk/c3t5d0

2. Take out and replace Replace the faulty Disk then fcmsutil replace_dsk
(Find the N port ID by performing a tail on syslog.log in /var/adm/syslog)

fcmsutil /dev/td0 replace_dsk 0xe0

3. Restore the LVM Header (note the use of rdsk):
# vgcfgrestore -n /dev/vg00 /dev/rdsk/c3t5d0

4. Activate the PV:
# pvchange -a y /dev/dsk/c3t5d0

5. Activate the VG:
# vgchange -a y /dev/vg01

6. vgsync (note, the VG may begin syncing after the vgchange):
# vgsync /dev/vg01

Hope that helps any poor souls looking for the answer.

Chris

Re: what is the best way of replacing disk that is "kind of" broken

Torsten. — Thu, 10 Apr 2008 16:58:15 GMT

Well done Chris.

This feature is called

LVM Online Disk Replacement (LVM OLR)

The guide is here:
http://docs.hp.com/en/7161/LVM_OLR_whitepaper.pdf

Hope this helps in future.

Re: what is the best way of replacing disk that is "kind of" broken

Kirk Reindl — Thu, 10 Apr 2008 17:02:05 GMT

Issue resolved