How to change a failed mirrored disk ?

[GNOM]

Hi,

I want to change a failed mirrored disk in HP-UX 11.11 environnement with LVM. The failed disk is mirrored on another disk on the second DS2300 disk enclosure with HP Mirror Disk softaware . I have 2 HP DS2300 Enclosures and each disk in the first enclosure is mirrored on another disk in the second enclosure. How can I change the disk Online without data lose?

Thanks!

[Torsten.]

Hi,

here is all you need to know:

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

[Gurumanickam]

.Replacing an attached strictly mirrored hot swap hard disk
DocId: LVMKBRC00006111 Updated: 6/11/04 9:00:00 AM
PROBLEM
A hot swap hard disk has to be replaced online. Can I just pull out the defect disk and insert a new one? What has to be considered, if not? Can you tell me the Hot Swap Procedure?
CONFIGURATION
HP-UX 10.X, 11.X
MirrorDisk-UX
hot swap hard disk
RESOLUTION
Replacing a bad mirrored disk by hot-swapping in a new disk can be a risky task, if the volume group is active and cannot be deactivated. If the disk is still attached to the volume group, the respective mirror copies of the affected logical volumes have to be reduced.
The following hot swap procedure is applicable for active volume groups and even if the physical volume is attached.
Prerequisites
1. All of the replaced disk's in-use extents, must belong to mirrored logical volumes which were created at least with the 'strict' option (-s).
2. The replacement disk must have enough space for the physical extents.

NOTE: The hotswap manual procedure will not update the disk driver's internal information to that of the replaced disk. One of the fields that could be different is the string specifying the vendor's name. This will not affect the behavior of the LVM.
WARNING: Only perform these steps during a maintainence window or when it is sure that the disk drive is capable of doing I/O. If the drive is faulty (ioscan for example would show NO_HW), lvreduce and vgreduce in steps 2 and 4 have the potential of hanging or failing, causing a possible unplanned outage. If there is a suspicion that the disk cannot do I/O, rather follow the steps in document KBAN00000347 LVM: Procedure for replacing an LVM disk in HP-UX 10.x and 11.x
A WORD TO PATCHES
Do not use LVM commands patches [PHCO_24809/PACHRDME/English] (HP-UX 11.11) or [PHCO_24645/PACHRDME/English] (HP-UX 11.0). These patches introduced behavior that can result in potential corruption of the Boot Data Reserved Area (BDRA) of a bootable volume group configured with alternate links.
Do not use HP-UX 11.0 mkboot patch [PHCO_20869/PACHRDME/English] , because it delivers a dynamically linked version of /sbin/mkboot, instead of a statically linked version of the file. Remove this patch and install the previous mkboot patch, [PHCO_19656/PACHRDME/English] .

Configuration Example
In the following typical example the vg01 contains the logical volumes 1, 2, and 3, which are strictly mirrored from disks on controller 2 to disks on controller 4:
Volume Description /dev/vg01/lvol1
/dev/vg01/lvol2 /dev/vg01/lvol2
/dev/vg01/lvol3
Primary /dev/dsk/c2t1d0 /dev/dsk/c2t2d0
Mirror Copy /dev/dsk/c4t1d0 /dev/dsk/c4t2d0
The disk /dev/dsk/c2t2d0 is defect and has to be replaced. According to the table above, /dev/vg01/lvol2 and /dev/vg01/lvol3 are affected. Because /dev/vg01/lvol2 spans over two disks, the other disk on the same mirror copy /dev/dsk/c2t1d0 has to be noted.
Hot Swap Procedure
1. Check which logical volumes are affected by this disk as shown above. Use

# vgdisplay -v vg01
to get all names of the logical volumes of volume group vg01 which contains /dev/dsk/c2t2d0. Use

# lvdisplay -v /dev/vg01/lvol1 | more
# lvdisplay -v /dev/vg01/lvol2 | more
# lvdisplay -v /dev/vg01/lvol3 | more
...
to get the distribution of the logical volumes.
NOTE: It is very important that all logical volumes are known which are affected by this disk. In addition, one has to know all physical volumes pv_path which are used by all these logical volumes. Otherwise, physical volume groups may become mixed up or data corruption may occur.
2. Reduce all mirrored logical volumes which are distributed on /dev/dsk/c2t2d0 writing down all physical volumes of the same mirror copy.
If the logical volume has one mirror copy:

# lvreduce -m 0 -A n /dev/vgXY/lvolZ all pv_paths of the same mirror
copy
or if the logical volume has two mirror copies:

# lvreduce -m 1 -A n /dev/vgXY/lvolZ all pv_paths of the same mirror
copy
In this Example:

# lvreduce -m 0 -A n /dev/vg01/lvol2 /dev/dsk/c2t1d0 /dev/dsk/c2t2d0
# lvreduce -m 0 -A n /dev/vg01/lvol3 /dev/dsk/c2t2d0
3. Check succesful mirror reduction by

# lvdisplay -v /dev/vg01/lvol2 | more
# lvdisplay -v /dev/vg01/lvol3 | more
4. Remove the physical volume from the volume group:

# vgreduce vg01 /dev/dsk/c2t2d0
Check with:

# vgdisplay -v vg01
5. Replace the disk.
6. Check connectivity with

# ioscan -fnC disk
7. Either initialize the raw disk device for use in a non-root volume group

# pvcreate /dev/rdsk/c2t2d0
or initialize the raw disk device for use in a root volume group (typically /dev/vg00, not applicable in this example):

# pvcreate -B /dev/rdsk/c0t6d0
8. Extend the volume group by adding the physical volume

# vgextend vg01 /dev/dsk/c2t2d0
10. Mirror all logical volumes, which have been reduced in step 2:

# lvextend -m 1 /dev/vgXY/lvolZ all pv_paths from above
or if the logical volume had two mirror copies:

# lvextend -m 2 /dev/vgXY/lvolZ all pv_paths from above
NOTE: It may not be necessary to name all pv_paths during this step if you are using physical volume groups and the allocation policy of the lvols is set to PVG-strict.
In this Example:

# lvextend -m 1 /dev/vg01/lvol2 /dev/dsk/c2t1d0 /dev/dsk/c2t2d0
# lvextend -m 1 /dev/vg01/lvol3 /dev/dsk/c2t2d0
11. Check that you have succesfully increased the number of mirror copies:

# lvdisplay -v /dev/vg01/lvol2 | more
# lvdisplay -v /dev/vg01/lvol3 | more
12. Finally, check the status of the volume group:

# vgdisplay -v vg01 | more
ALT KEYWORDS
ULVM
KBRC00009115 (document deleted)


9. Skip the following step if the disk is NOT a root disk.
For a root disk perform the following commands:
Use mkboot to place boot utilities and an AUTO file in the boot LIF area

# mkboot -l /dev/rdsk/c0t6d0
# mkboot -a "hpux" /dev/rdsk/c0t6d0
Or, use the -lq option to allow the system to boot in the event that one of the disks is unavailable, resulting in a loss of quorum.
# mkboot -a "hpux - lq" /dev/rdsk/c0t6d0
NOTE: Use the -lq option when the root volume group contains only 2 disks. If more than 2 disks are in the root volume group, the -lq option is not needed as quorum will be maintained even if 1 disk fails.
For HP-UX 10.20 (not updated to) and 11.X only:

# lvlnboot -r /dev/vg00/lvol3 /dev/vg00
# lvlnboot -b /dev/vg00/lvol1 /dev/vg00
# lvlnboot -s /dev/vg00/lvol2 /dev/vg00
# lvlnboot -d /dev/vg00/lvol2 /dev/vg00
# lvlnboot -v
With LVM commands cumulative patches newer than [PHCO_24809/PACHRDME/English] (11.11) or [PHCO_24645/PACHRDME/English] (11.0) and configured alternate links (check with /usr/sbin/lvlnboot -v | grep "alternate link") use additionally
# lvlnboot -R
to recover missing links immediately.
For HP-UX 10.01, 10.10, and updates to 10.20 only:

# lvlnboot -r /dev/vg00/lvol1 /dev/vg00
# lvlnboot -s /dev/vg00/lvol2 /dev/vg00
# lvlnboot -d /dev/vg00/lvol2 /dev/vg00
# lvlnboot -v
If you want to reinstall the Offline Diagnostics ODE on this disk please refer to the document LVMKBRC00005103

[GNOM]

Thanks for your help.

Otherwise I don't really know if I have to change the disks because it's still CLAIMED an data is still available on the disk but the error in the mail recommend to replace the disk.
Here is an amonut of the mails :

Summary:
Disk at hardware path 0/4/1/0.8.0 : Hardware failure


Description of Error:

The device driver has received an unspecified hardware error from the
device.

Probable Cause / Recommended Action:

The device has experienced a hardware failure. Contact your HP support
representative to have the device checked.

Alternatively, if messages corresponding to this error appear in the log
for more than one device on the same SCSI bus, the device adapter has
experienced a hardware failure. Replace it.




Summary:
Disk at hardware path 0/4/1/0.9.0 : A SMART event has occurred.


Description of Error:

The drive has predicted that it will fail in the near future. Contact
your HP support representative as soon as possible.

Probable Cause / Recommended Action:

A SMART event has occurred. The drive has detected a serious condition in
itself that indicates it will soon fail. Contact your HP support
representative as soon as possible. Back up the data on the drive now!

[Pete Randall]

It's spelled out pretty clearly in the message:

"The drive has predicted that it will fail in the near future. Contact
your HP support representative as soon as possible.

Probable Cause / Recommended Action:

A SMART event has occurred. The drive has detected a serious condition in
itself that indicates it will soon fail."

I would suggest you do what it says because it will most likely fail in the near future. If you have a support contract, just call HP and give them the message you're receiving and they'll replace the drive.


Pete

[Mridul Shrivastava]

You must have following OLAR patches installed on the system so this disk can be replaced online:
Patches PHKL_31216 and PHCO_30698 (or superseding patches)

Otherwise ur hardware may support hot swap but from LVM perspective if these patches are not installed and u remove the disk online .. if there is some io going on the disk at that time.. the this may cause further problems....

It's good to have these patches installed to enable online replacement.