Mirrored Disk Failing

[Dan Powers]

Hello Everyone,

I have a disk failing in a SC10 mirrored configuration. It is the primary disk not the mirror. It is part of a 3 disk VG. c0t13d0 is the failing disk and c4t14d0 is the mirror. The error I am receiving is:
INFORMATIONAL(RC) - 3:33 AM, 04/12/04 - Pdev 0/0/1/0.13.0:OEM(773)
DMVS18DIBM:SCSI disk drive has correctable errors : Type 2: Occurrences 27

The process I'm intending to use is the following:

-vgcfgbackup -f /tmp/vg01.bu vg01
-replace the disk (hot swappable)
-vgcfgrestore -n /dev/vg01 /dev/rdsk/c0t13d0
-vgchange -a y /dev/vg01
-vgsync /dev/vg01

I believe this should work. The person who built the system keeps telling me the LV's are mirrored and not the disks specifically. If this is the case would the above process still work with adding an lvsync for each LV located on the primary disk?

Thanks in advance!

[Todd McDaniel_1 · Accepted solution]

Lvsync is an extra step in my mind. but you are more than welcom to do it... BTW, when you do vgsync it will show you syncing of the Lvols.

If your VG/LV is mirrored, Im not sure what your adivsor is saying about the disks not being mirrored... The disks mirror each other in the LVOL/VG.

You should be fine with what you have... however, I would umount the FS if you can and vgchange -an vgXX

If the disk is not truly dead it has a possibility of panicing your box, IF you pull the disk.

do a diskinfo and pvdisplay to see if it is dead or dying... IF only dying, I would deactivatge the VG before pulling the disk.

[Dan Powers]

Thanks for the quick response! So it doesn't matter what disk fails. The system will sync the mirror to the primary or vice-versa?

[A. Clay Stephenson]

Your person is correct, mirroring is done at the logical volume level therefore you need to do a little more work.

1) Do a vgdisplay -v /dev/vg01 and note all the LVOL's

2) Do an lvdisplay -v /dev/vg01/lvolx for each LVOL and make sure that every LVOL is mirrored and that every extent on the GOOD mirror is current. You don't want to be in a situation where you have state extents on both disks.

I suppose that you are doing a vgcfgbackup to /tmp just for a bit of extra security. Well, there is probably a cleanup script in init.d that will remove files in /tmp and /var/tmp so you might consider a different location.

3) Yank (gently) the old disk out and let it spin down while still partially in the bay. This will prevent possible further damage.
Now wait two minutes or so until the system knows that this disk is dead before inserting the new disk.

4) Now your procedure is fine beginning with vgcfgrestore. The vgsync can take a few tens of minutes so don't get impatient. The vgsync syncs all the LVOL's in the VG.

[Sanjay_6]

Hi Dan,

The steps would work as long as the primary and the mirror copy are on seperate luns. Do a lvdisplay -v for all lvs in the vg to make sure the two copies are on seperate disks. If it is do, doing the steps you have outlined should work.

Hope this helps.

Regds

[Michael_114]

Check in /var/adm/syslog/syslog.log to see that LVM has detected the removal of the disk. If it hasn't detected it, it will not let you run through the recovery proccess because it will think that the disk was not replaced and may give errors about the disk belonging to a VG already.

[Dan Powers]

A quick question about two recommended steps...
1) Do a vgdisplay -v /dev/vg01 and note all the LVOL's

2) Do an lvdisplay -v /dev/vg01/lvolx for each LVOL and make sure that every LVOL is mirrored and that every extent on the GOOD mirror is current. You don't want to be in a situation where you have state extents on both disks.

My question is do I need to be concerned with the lvols on all disks in the VG or just the ones located on the failing disk?

Thanks!

[A. Clay Stephenson]

I trying to get you to realize that you must first verify that each LVOL is indeed mirrored. You could yank the drive out to discover that , oops, lvol12 was not mirrored. For unmirrored LVOL's (on the failing disk) you have two choices: 1) Mirror it 2) Back it up.

[Dan Powers]

I'd like to thank everyone that participated in this thread. I did the drive replacement on Saturday and everything went fine. Thanks again!

Dan