what is your best experienced way to replace a failed disk in vg00?

[stephen peng]

guys,
before I get HP-When_Good_Disks_Go_Bad_WP, when I met a failed disk in vg00,which appeared to be unavailable in vgdisplay -v, I used lvreduce -m 0 /dev/vg00/lvol* /dev/dsk/c*t*d* to reduce mirrored copys of LVs on the failed disk, which would cause a lot of time(10 minutes plus for each lv), and then refer to the manual, I used lvreduce -m 0 -k /dev/vg00/lvol* /dev/dsk/c*t*d* key to reduce mirrored copys, it looked alright, not causing so much time before, but yesterday, it turned out to be just time consuming like not using -k.
so what is YOUR way to deal with such situation, RAPIDLY?

thanks a lot!

[Bijeesh · Accepted solution]

>>what is your best experienced way to replace a failed disk in vg00?<<
If you want to replace the hard disk,no need to reduce the mirror from failed disk.
After replacing the disk, vgsync command sync for all LVs.
Refer the flow chart for complete steps on that manual.

But if ONLY want to remove the mirror copy, from the unavailable disk, the method mentioned in the manual is safe.ie with -k option. But at this time you need to reduce the the vg and update the lvmtab using vgscan to reflect remaining PV for that VG.

[Torsten.]

In more than 10 years I did such a mirror reduce once or twice, always with problems, but I did some hundreds of disk replacements successfully.

[TwoProc]

Yep, vgsync is your buddy. If it is a boot disk, put in the new disk, "pvcreate -B" it, run your "mkboot -a" command first, and then do vgsync command, as vgsync is not going to restore your bootable areas for you.
If the disk your replacing isn't a boot disk, you just run vgsync for the vg you're interested in. I think if you run it with no arguments it will find what needs syncing for you, but its been a while, so I'm not sure on that.

[vishnu.khandare]

Hi Stephen,

Have U indentifies which one is ur failed disk,
After failed disk has been replaced (assuming the disk is c*t*d*):

vgcfgrestore -n /dev/vg00 /dev/rdsk/c*t*d*

mkboot /dev/rdsk/c*t*d*

mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c*t*d*

vgchange -a y /dev/vg00

lvlnboot -Rv /dev/vg00

vgsync /dev/vg00


Now,

1) If the disk is completely "dead", such as if you run ioscan and status is "no_hw" then you can hot swap the disk online.

2) However in circumstances where the disk has not fully failed please do one of the following to avoid data corruption :

a) reduce mirror before replacing the disk
b) deactivate VG before replacing the disk
c) shutdown system to replace the disk



If you have to rebuild the mirror:

Say /dev/dsk/c*t*d* is the root disk and /dev/dsk/c*t*d* is the disk you want to make a mirror of.

1. Create a bootable LVM disk to be used for the mirror.
pvcreate -B /dev/rdsk/c*t*d*
2. Add this disk to the current root volume group.
vgextend /dev/vg00 /dev/dsk/c*t*d*
3. Make the new disk a boot disk.
mkboot -l /dev/rdsk/c*t*d*
4. Copy the correct AUTO file into the new LIF area.
mkboot -a "hpux -lq (;0)/vmunix" /dev/rdsk/c*t*d*
5. Mirror the boot, root and primary swap logical volumes to the new
bootable disk. Ensure that all devices in vg00, such as /usr, /swap,
etc., are mirrored.
6. Update the boot information contained in the BDRA for the mirror
copies of boot, primary swap and root.
/usr/sbin/lvlnboot -b /dev/vg00/lvol1
/usr/sbin/lvlnboot -s /dev/vg00/lvol2
/usr/sbin/lvlnboot -r /dev/vg00/lvol3
7. Check if the BDRA is correct.
/usr/sbin/lvlnboot -R /dev/vg00
8. Verify that the mirrors were properly created.
lvlnboot -v /dev/vg00
The output of this command is shown in a display like the following:
Boot Definitions for Volume Group /dev/vg00:
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c*t*d* (*/0/0/*/0.*.0) -- Boot Disk
/dev/dsk/c*t*d* (*/0/*/0/0/*/*.*.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Root: lvol3 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Swap: lvol2 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Dump: lvol2 on: /dev/dsk/c*t*d*, 0

vgdisplay -v vg00

Then lvextend for all other lvols:
By
lvexten -m 1 /dev/vg00/lvolX /dev/dsk/c*t*d*


Hope this solves ur query
Don't forget to assign points.

Regards
Vishnu Khandare

[stephen peng]

guys, what I wanted to know was how to safely and rapidly replace a failed disk in vg00, I assumed lvreduce then vgreduce was the safest way, maybe I was wrong then.
Bijeesh,
I never tried your method to replace a failed disk in vg00, and it seemed that you miss pvcreate -B.
Torsten,
>>but I did some hundreds of disk replacements successfully.
you did not mentioned your way.
TwoProc,
like I said, I never tried this way, and don't know what it would turn out to be.
Khandare,
I knew this method,but failed once,and I think the order of all the commands were important, and you miss pvcreate -B I think.

anyway,thank you for your reply.

[SoorajCleris]

Hi,

Do you have the doc , "When Good Disks Go Bad" ?

Its all there! You may kindly download it and refer.

regards,
Sooraj

[Bijeesh]

Hi,
After replacing the disk follow the below commands.Here pvcreate -B is not required.
#vgcfgrestore -n /dev/vg00 /dev/rdsk/c?t?d?
#mkboot /dev/rdsk/c?t?d?
#mkboot -a "hpux -lq" /dev/rdsk/c?t?d?
#vgchange -a y /dev/vg00
#lvlnboot -R /dev/vg00
#vgsync /dev/vg00

And for integrity server first create partitions on the disk using idisk command.
Refer the manual properly.

[stephen peng]

Bijeeshï¼
sorry about my mistake, pvcreate should not be used in such situation, I remind.

[chris huys_4]

Hi,

In enterprise environments, the primary objective should always be, will the procedure work "safely", other "objectives", "like time", should not matter and ever moreso, if everything can be done "online". Offcourse you should also be able to "estimate more or less accurate ,onbeforehand, how long the procedure will take " to get executed.

"Safely", means that the one executing the procedure is confident that the procedure will work as he expects, each and every time.

"estimate more or less accurate, onbeforehand, how long the procedure will take" "in a particular situation", is something that can only be "learned" from "experience" over the years.

In most situations, both the "vgcfgrestore" procedure and the "lvreduce" procedure can be used.

Learn them both on a testsystem, see what works best for you, and if there is a choice between using the 2 methods, use the one that you feel most confident with, not the one who will save 2 minutes but were you feel less confident in.

I, personally, until know, always used, in production environments, if possible, the "lvreduce -k" procedure.

Greetz,
Chris