what is your best experienced way to replace a failed disk in vg00?

[stephen peng]

guys,
before I get HP-When_Good_Disks_Go_Bad_WP, when I met a failed disk in vg00,which appeared to be unavailable in vgdisplay -v, I used lvreduce -m 0 /dev/vg00/lvol* /dev/dsk/c*t*d* to reduce mirrored copys of LVs on the failed disk, which would cause a lot of time(10 minutes plus for each lv), and then refer to the manual, I used lvreduce -m 0 -k /dev/vg00/lvol* /dev/dsk/c*t*d* key to reduce mirrored copys, it looked alright, not causing so much time before, but yesterday, it turned out to be just time consuming like not using -k.
so what is YOUR way to deal with such situation, RAPIDLY?

thanks a lot!

[Bijeesh · Accepted solution]

>>what is your best experienced way to replace a failed disk in vg00?<<
If you want to replace the hard disk,no need to reduce the mirror from failed disk.
After replacing the disk, vgsync command sync for all LVs.
Refer the flow chart for complete steps on that manual.

But if ONLY want to remove the mirror copy, from the unavailable disk, the method mentioned in the manual is safe.ie with -k option. But at this time you need to reduce the the vg and update the lvmtab using vgscan to reflect remaining PV for that VG.

[Torsten.]

In more than 10 years I did such a mirror reduce once or twice, always with problems, but I did some hundreds of disk replacements successfully.

[TwoProc]

Yep, vgsync is your buddy. If it is a boot disk, put in the new disk, "pvcreate -B" it, run your "mkboot -a" command first, and then do vgsync command, as vgsync is not going to restore your bootable areas for you.
If the disk your replacing isn't a boot disk, you just run vgsync for the vg you're interested in. I think if you run it with no arguments it will find what needs syncing for you, but its been a while, so I'm not sure on that.

[vishnu.khandare]

Hi Stephen,

Have U indentifies which one is ur failed disk,
After failed disk has been replaced (assuming the disk is c*t*d*):

vgcfgrestore -n /dev/vg00 /dev/rdsk/c*t*d*

mkboot /dev/rdsk/c*t*d*

mkboot -a "hpux -lq (;0) /stand/vmunix" /dev/rdsk/c*t*d*

vgchange -a y /dev/vg00

lvlnboot -Rv /dev/vg00

vgsync /dev/vg00


Now,

1) If the disk is completely "dead", such as if you run ioscan and status is "no_hw" then you can hot swap the disk online.

2) However in circumstances where the disk has not fully failed please do one of the following to avoid data corruption :

a) reduce mirror before replacing the disk
b) deactivate VG before replacing the disk
c) shutdown system to replace the disk



If you have to rebuild the mirror:

Say /dev/dsk/c*t*d* is the root disk and /dev/dsk/c*t*d* is the disk you want to make a mirror of.

1. Create a bootable LVM disk to be used for the mirror.
pvcreate -B /dev/rdsk/c*t*d*
2. Add this disk to the current root volume group.
vgextend /dev/vg00 /dev/dsk/c*t*d*
3. Make the new disk a boot disk.
mkboot -l /dev/rdsk/c*t*d*
4. Copy the correct AUTO file into the new LIF area.
mkboot -a "hpux -lq (;0)/vmunix" /dev/rdsk/c*t*d*
5. Mirror the boot, root and primary swap logical volumes to the new
bootable disk. Ensure that all devices in vg00, such as /usr, /swap,
etc., are mirrored.
6. Update the boot information contained in the BDRA for the mirror
copies of boot, primary swap and root.
/usr/sbin/lvlnboot -b /dev/vg00/lvol1
/usr/sbin/lvlnboot -s /dev/vg00/lvol2
/usr/sbin/lvlnboot -r /dev/vg00/lvol3
7. Check if the BDRA is correct.
/usr/sbin/lvlnboot -R /dev/vg00
8. Verify that the mirrors were properly created.
lvlnboot -v /dev/vg00
The output of this command is shown in a display like the following:
Boot Definitions for Volume Group /dev/vg00:
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c*t*d* (*/0/0/*/0.*.0) -- Boot Disk
/dev/dsk/c*t*d* (*/0/*/0/0/*/*.*.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Root: lvol3 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Swap: lvol2 on: /dev/dsk/c*t*d*
/dev/dsk/c*t*d*
Dump: lvol2 on: /dev/dsk/c*t*d*, 0

vgdisplay -v vg00

Then lvextend for all other lvols:
By
lvexten -m 1 /dev/vg00/lvolX /dev/dsk/c*t*d*


Hope this solves ur query
Don't forget to assign points.

Regards
Vishnu Khandare

[stephen peng]

guys, what I wanted to know was how to safely and rapidly replace a failed disk in vg00, I assumed lvreduce then vgreduce was the safest way, maybe I was wrong then.
Bijeesh,
I never tried your method to replace a failed disk in vg00, and it seemed that you miss pvcreate -B.
Torsten,
>>but I did some hundreds of disk replacements successfully.
you did not mentioned your way.
TwoProc,
like I said, I never tried this way, and don't know what it would turn out to be.
Khandare,
I knew this method,but failed once,and I think the order of all the commands were important, and you miss pvcreate -B I think.

anyway,thank you for your reply.

[SoorajCleris]

Hi,

Do you have the doc , "When Good Disks Go Bad" ?

Its all there! You may kindly download it and refer.

regards,
Sooraj

[Bijeesh]

Hi,
After replacing the disk follow the below commands.Here pvcreate -B is not required.
#vgcfgrestore -n /dev/vg00 /dev/rdsk/c?t?d?
#mkboot /dev/rdsk/c?t?d?
#mkboot -a "hpux -lq" /dev/rdsk/c?t?d?
#vgchange -a y /dev/vg00
#lvlnboot -R /dev/vg00
#vgsync /dev/vg00

And for integrity server first create partitions on the disk using idisk command.
Refer the manual properly.

[stephen peng]

Bijeeshï¼
sorry about my mistake, pvcreate should not be used in such situation, I remind.

[chris huys_4]

Hi,

In enterprise environments, the primary objective should always be, will the procedure work "safely", other "objectives", "like time", should not matter and ever moreso, if everything can be done "online". Offcourse you should also be able to "estimate more or less accurate ,onbeforehand, how long the procedure will take " to get executed.

"Safely", means that the one executing the procedure is confident that the procedure will work as he expects, each and every time.

"estimate more or less accurate, onbeforehand, how long the procedure will take" "in a particular situation", is something that can only be "learned" from "experience" over the years.

In most situations, both the "vgcfgrestore" procedure and the "lvreduce" procedure can be used.

Learn them both on a testsystem, see what works best for you, and if there is a choice between using the 2 methods, use the one that you feel most confident with, not the one who will save 2 minutes but were you feel less confident in.

I, personally, until know, always used, in production environments, if possible, the "lvreduce -k" procedure.

Greetz,
Chris

[Subentu]

1. Remove the failed disk and Replace with new one (Can be Done Online)

2. Recreate device files and ensure the new disk is getting detected in ioscan.
# insf -e
# ioscan -fnC disk

3. Create partitions on the new disk
# vi /tmp/idf
3
EFI 500MB
HPUX 100%
HPSP 400MB
:wq!

4. Verify the Partition table on new disk.
# idisk /dev/rdsk/c3t6d0

5. Run "insf and ioscan " again and verify device files for partitions are created properly.
# insf -e
# ioscan -fnC disk

6. Populate the /efi/hpux directory in the EFI partition
# mkboot -e -l /dev/rdsk/c3t6d0

7. Update the AUTO file contents on disks.
# vi /tmp/auto
boot vmunix â lq
:wq!

# efi_cp â d /dev/rdsk/cXtXdXs1 /tmp/auto /efi/hpux/auto >>>> Replace cXtXdX with device file of Primary disk.

# efi_cp â d /dev/rdsk/c3t6d0s1 /tmp/auto /efi/hpux/auto >>>> New disk

8. Verify the contents of the system partition.
# efi_ls â d /dev/rdsk/c3t6d0s1

9. Verify the contents of the auto file in the system partition.
# efi_cp â d /dev/rdsk/c3t6d0s1 â u /efi/hpux/auto /tmp/auto

10. Restore LVM headers to the new disk.
# vgcfgrestore â n /dev/vg00 /dev/rdsk/c3t6d0s2

11. Reactivate the volume group to recognize the recovered disk.
# vgchange â a y /dev/vg00

12. Resynchronize mirrors.
# vgsync vg00

[stephen peng]

Chris,
I agree with your opinion, and just like you, all the time I used "lvreduce -m 0" then "lvreduce -m 0 -k" to replace a failed disk in vg00, and I do feel confident about this way. but, frankly, it cause me more than one hour to finish the whole lvreduce procedure, not liked you said, "two minutes", especially when the failed disk appeared to be "unavailable" in vgdisplay.
maybe I am just looking for some "confident" way from other experts.

regards
stephen

[Shibin_2]

I faced a similar situation in a critical production environment. Some requested to do the lvreduce, but I was confident over the pvchange command. System was running with HP-UX 11.11.

Here is the step, I took to replace the failed disk of rootvg.

==================

Detach LVM and all information from the disk to LVM
#pvchange -a N

# pvcreate -f -B /dev/rdsk/
# mkboot -l /dev/rdsk/
# mkboot -a "hpux -lq" /dev/rdsk/
# vgcfgrestore -n vg00 /dev/rdsk/
=> recovers LVM info to disk
# pvchange -a y /dev/dsk/
=> enables new root disk
# vgsync vg00
=> Synchronize the VG and LVs, this will take awhile
Both disks are now in sync.
# lvlnboot -Rv
=> Update the boot information on mirror disk.
#mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p HPUX -p LABEL -p AUTO
=> Update the LIF details to the mirrored disk.

[stephen peng]

Shibinï¼
Not every 11.11 version support pvchange . another issue, was pvcreate -f -B necessary?

regards
stephen

[chris huys_4]

Hi Stephen,

Ah its that system. You should have mentioned that the system never gets patched up and still doesnt have the 3 OLR patches installed as one of the conditions to come up with another procedure. ;)

Indeed in that case, lvreduce is still the safest method, although slow in some cases.

In all other cases, shibin's procedure(s) looks like a good replacement procedure for the "lvreduce" procedure.

Greetz,
Chris

[vishnu.khandare]

Hi Stephen,

==>Not every 11.11 version support pvchange . another issue, was pvcreate -f -B necessary?

Answer==> -f option with pvcreate is force the disk to pvcreate, -B option is to make disk bootable.
Some time pvcreate -B does not work, it gives error, that time we need to use -f option.

Regards
Vishnu Khandare

[stephen peng]

Chris,
maybe lvreduce is our safest method, maybe others get their own safest method,maybe I am anticipating that some one telling me "go! it is OK!"

regards
stephen

[Shibin_2]

Hi Stephen,

Yes, you are right. Not all HP-UX 11.11 support pvchange command. You need to have necessary patch levels for this.

The machine I was working on it was MCOE.

Btw, that is your root disk mirror. So, you need to make it as bootable with -f -B option ( Vishnu explained it well ).