How to replace a Primary Drive (mirrored) on an rx1600 with UX11.23

[nibble]

Guys, there has been a lot of posting for replacing a faulty disk (secondary) but i havent tried replacing a root disk (primary to be replaced) and on an rx1600 -- details below:

Server model & OS : ia64, hp server rx1600, HPUX B.11.23 64 bit
Hard drive: 2 x 36Gb, hot plugged, UX mirrored ( Drive A - primary & Drive B - secondary)
Error: Primary drive Drive A, has some stales on its LVs, pvdisplay can report that its available so
its "Attached", drives are claimed/can be found in ioscan, vgdisplay shows no errors, except "available/stale" status n the 3 affective LVs (there are 10 LVs)

A. Its clear that its hot-swappable, but i havent tried replacing an rx1600. is it safe to replace the primary drive online?
B. If i dont have the exact 36Gb, can i replace it with a higher capacity?
C. If i can replace it online, this is the procedure i found...Im not sure if this is applicable if the drive to be replaced is the Primary Mirror

assuming: c0t0d0 primary, c0t1d0 secondary
- Reduce the lv
# lvreduce -m 0 -A n /dev/vg00/lvolX /dev/dsk/c0t0d0s2

- replace the drive ONLINE

- Verify if the new drive is claimed
# ioscan -f /dev/dsk/c0t0d0

- verify the disk partitions of the secondary
# diskinfo /dev/rdsk/c0t1d0s1|grep size
# diskinfo /dev/rdsk/c0t1d0s3|grep size

- create the partition description file:
# vi /tmp/partitionfile
3
EFI 500MB <-- where will i get these values
HPUX 100%
HPSP 400MB

- Clear the previously partition configuration
# idisk -R /tmp/partitionfile /dev/rdsk/c0t0d0sZ

- Create the disk partitions
# idisk -wf /tmp/partitionfile /dev/rdsk/c0t0d0s2

- Create the new device files for the new partitions
# insf -e -Cdisk

- verify the partitiontable
# idisk /dev/rdsk/c0t0d0

- initialize the FAT filesystem on the EFI partition
# efi_fsinit -d /dev/rdsk/c0t0d0s1

-Use mkboot to format the EFI partition (s1) and populate it with the EFI files below /usr/lib/efi/
and to format the LIF volume (part of s2) and populate it with the LIF files (ISL, AUTO, HPUX, LABEL)
below /usr/lib/uxbootlf:

# mkboot -e -l /dev/rdsk/c0t0d0
# efi_ls -d /dev/rdsk/c0t0d0s1 (to check EFI)
# lifls -l /dev/rdsk/c0t0d02 (to check LIF)


- Check the content of AUTO file on EFI partition on primary disk:
# efi_cp -d /dev/rdsk/c0t0d0Zs1 -u /EFI/HPUX/AUTO /tmp/prim
#cat /tmp/prim
NOTE: Specify the -lq option if prefer that your system boots up without interruption in case of a
disk failure:
# mkboot -a "boot vmunix -lq" /dev/rdsk/c0t0d0
# mkboot -a "boot vmunix -lq" /dev/rdsk/c0t1d0

- Check the content of AUTO file on EFI partition on new mirror disk:
# efi_cp -d /dev/rdsk/c0t0d0s1 -u /EFI/HPUX/AUTO /tmp/mirror
# cat /tmp/mirror

- Copy the HP service partition (UX 11.23 only): (skip this, if you don’t have a service partition) <-- HOW will i know if i have a service partition????
# dd if=/dev/rdsk/c0t1d0s3 of=/dev/rdsk/c0t0d0s3 bs=1024k

- Now the disk is configured as boot devices.
Now we have to put the hpux info back.

- Restore the LVM configuration/headers onto the replaced disk from your backup
#vgcfgrestore –n vg00 /dev/rdsk/c0t0d0s2

- Attach the new disk to the active volume group with the vgchange command
#vgchange –a y vg00

- Extend the mirrors back onto the replaced disk.
#lvextend –m 1 /dev/vg00/lvolX /dev/dsk/c0t0d0s2 <--- CAN I do this considering that it was the primary that i replaced?

4. As much as possible, No down time

Guys, any help will be highly appreciated

[Torsten.]

There is no difference between primary or secondary - each part of the mirror is equal.

For correct steps see

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

[Theodore Pardike]

Nibble,

We had a primary disk fail in our rx4640. What a pain. We went through all of the procedures and could never get it to boot off of the primary drive (though everything looked good, except when we rebooted). After 3 attempts and an 8 hour day (on a Sunday) on the phone with HP (with 4 different technicians) we finally got it resolved.
We tried so many different things, but I am thinking that the ultimate solution was to run the following command after the disk was replaced:
setboot -p 0/1/1/0.1.0
(Make sure you put the hardware address of your primary drive in there.)
This let's the EFI know that there is a new primary disk.

Also, make sure you are at the proper patch level to support LVM Online Disk Replacement (LVM OLR).
http://docs.hp.com/en/7161/LVM_OLR_whitepaper.pdf

When your done, make sure you run the following command to verify that you are booting off of the disk you think you are:
grep "Boot device" /var/adm/syslog/syslog.log

For what it's worth, enclosed are my notes on the procedure.
Use at your own risk.
If nothing else, at least it will give you some ideas.

Ted

[Siju Vadakkan]

A. Its clear that its hot-swappable, but i havent tried replacing an rx1600. is it safe to replace the primary drive online?
You can replace the disk if it is hot swappable and properly mirrored. It doesnâ t matter whether is it primary or secondary.
B. If i dont have the exact 36Gb, can i replace it with a higher capacity?
You can replace a disk with higher capacity


1. Deactivative Physical Volume
#pvchange â a n
2. replace the failed disk
3. run #ioscan â fnC disk

4. Create the EFI and OS partitions using an IPF partition description file
5.
# vi /tmp/idf
2
EFI 100MB
HPUX 100%
# idisk â f /tmp/idf â w /dev/rdsk/c0t1d0
6. Populate the /efi/hpux directory in the new EFI partition
# mkboot â e â l /dev/rdsk/c0t1d0
7. Change the AUTO file on both boot disks to hpux â lq so the host can boot without quorum
# vi /tmp/efi.auto
boot vmunix â lq
# efi_cp â d /dev/rdsk/c0t1d0s1 /tmp/efi.auto /efi/hpux/auto
8. activate the PV #pvchange â a y
9. Restore LVM headers to the disk
# vgcfgrestore â n vg00 /dev/rdsk/c0t1d0s2
10. Reactivate the VG to recognize the recovered disk
# vgchange â a y vg00
11. Resynchronize mirrors
# vgsync vg00

[nibble]

Hi Theodore,
I did not see the lvreduce nor the vgreduce in your document. Can i assume that its being done before the steps that you send?
The primary drive is not dead, its completely recognized by the machine and it is active and attached. its just that there are few LEs that are stale which a sign of a bad block.

[Mridul Shrivastava]

You may not need to do lvreduce if you have LVM OLR patch installed. or try pvchange command if it works. If it is there then you have to just detach the PV with stale extents using "pvchange -a n "

After this disk could be replaced safely since there won't be any request sent to that drive. Once disk is replaced attach it again using the "pvchange -a y "
option.

So you won't have to do lvextend again only vgcfgrestore will update the information on the disk. However the other steps will remain the same since it is a boot disk.

[Torsten.]

.


IMHO most of the problems during a disk replacement are created by users doesn't follow/using the correct procedure or just trying.
An example of a wrong step is to vgextend the new disk without remove it from the config earlier or use an old procedure for 11.21 or 11.22 (only 2 partitions on the disk) etc.

Follow the document I posted above.

[nibble]

Yes Torsten. I really appreciate that. I have to be extremely careful cause this is a production server. Thanks. Thats why Im gathering much information as possible. we dont have HP support so, i have to do it myself.

[Theodore Pardike]

Nibble,

Without a support contract I would be very nervous as well.
I think you need to determine the following before proceeding:
1. Is your drive really hot-swappable (the one in our rx-4640 was).
2. Are you at the correct patch level to support LVM OLR.

Our primary disk completely failed and our mirror took over for it.
In your circumstance, I think I would boot from the secondary and follow the instructions I provided (you should be able to pull the primary after booting from the secondary), assuming you met the requirements above.
The document I provided came from HP, with a few embellishments from me. Also, review some of the links at the bottom for reference.
The biggest problem we had (and what hung up the technicians), was that the EFI needs to know that the disk has been replaced. If this doesn't happen, then everything will look like it's working, until you try to boot from the drive.
Ultimately, HP had me go into the EFI and rewrite the boot entry for the primary disk.
Step 3a in my instructions is an assumption on part. Theoretically, that step should rewrite the boot entry for the primary drive without having to go to the EFI shell to do it.

In your postion, I would do the following:
1. Make 2 recovery tapes before doing anything.
2. Wait for more comments from the forum.
3. Make sure you can boot from the Secondary before proceeding.

When I did mine, I was amazed that the whole procedure wasn't more straightforward than what it was.

Also, like I said, when your done, verify that it is truly booting off of the primary. Because if your not watching carefully, everything looks OK when it really isn't.

Ted

[nibble]

theodore, just a quick question on how to verify if i have the appropriate LVM OLR version? I know swlist can do it, but which package/product or fileset is it?

Ive got this latest one in my test machine (11.11 - hp9000),

# swlist -l product|grep -i lvm
LVM B.11.11 LVM
PHCO_35955 1.0 LVM commands cumulative patch
PHKL_30472 1.0 VxVM,EMC,PM,vPar,slpq1,KRS,CDUMP,EVA,LVM,VM
PHKL_35970 1.0 LVM Cumulative Patch