some extents current even after disk replaced

[Tim Nelson]

Scenario: A disk in a mirrored set has failed. I replaced the disk, ran vgcfgrestore and vgchange. When I look at a pvdisplay for the new disk or lvdisplay for the lvol I see some extents stale and some current for this new blank disk. If I run lvsync or vgsync will the extents on the new disk that are marked as currently really get re-sync'd from the good extents on the other disk ? This poses another question. If the disk was dead why is lvdisplay or vgdisplay still reporting some extents as current ? Any help/explanation would be welcomed. My first safe approach would be to reduce this mirrored set and re-mirror it. This would be the better to be safe than sorry but I would rather understand why things are the way they are. Attached is an example of the lvdisp.ay. The c3t15 disk was dead as a doornob. This is the lvdisplay after the disk replacement. Thank you !!

[Tim Nelson]

Here is some additional info. While I was researching this Oracle generated an I/O error. This is interesting as the still active mirror is working fine. I ended up reducing the new disk out of the mirror set and readded it. I would be interested if anyone else has seen this type of behavior. Thanks in advance.

[A. Clay Stephenson · Accepted solution]

Hi Tim:

You are not in bad shape and a vgsync (or an lvsync if you prefer will fix you). The reason that some of the extents are marked current is that they were able to updated using the Mirror Write Cache when you did the vgchange. Note that your vgdisplay show:
Consistency Recovery MWC

The current extents on both disks are indeed current and a vgsync will correctly sync the stale extents to the current and take no action on the current extents.

Hope this clears it up, Clay

[John Palmer]

Hi,

I've seen this happen a couple of times before. You MUST follow the correct procedure to replace a failed LVM mirrored disk.

I believe that the system only knows that extents are stale when it tries to tries to update them and the write to that disk fails. Extents that have not been updated are still current.

If you replace a disk without ensuring that either:
- all extents are marked stale or
- all extents are removed from the failed disk
then the system will continue to read current extents from the new disk when in fact these contain absolute garbage. This will result in filesystem and data corruption.

Document ID KBAN00000347 describes the procedure for replacing an LVM disk and LVMKBRC00006111 the procedure for replacing a hot swap disk. If you follow the former for replacing a hot swap disk, then you can cause the above corruption if you don't reboot the server (which is one of the steps because it assumes that the server has to be powered off to replace the disk). Is this what you did? The reboot and reactivation of the volume group causes all extents on the failed disk to be marked stale - so no problem!

The hot swap procedure requires you to remove ALL extents from the failed disk with lvreduce before swapping the disk. You have the choice of leaving it in the VG empty and vgcfgrestoring after replacement or removing it from the VG altogether, replacing it then pvcreateing the replacement and vgextend.

Hope this helps,

Regards,
John

[Sridhar Bhaskarla]

Tim,

I saw this behaviour before.

Replace the disk
Do vgcfgrestore
Do vgchange -a y

and you see stale/current entries. Did you compare it against the previous lvdisplay output?

I feel your vgcfgrestore is the culprit here. That might happen if the vgcfgbackup was run after the disk failure.

I would safely remove the mirrors and recreate them. I do not believe in running vgsync. CLEAN.

[Tim Nelson]

Yep.. This boiled down to a poor procedure when replacing a failed disk. Never assume. If and lvdisplay shows any current extents even on a bad disk you must get that disk out of the lvm config what ever way possible before replacing the failed mech.
Thanks to all !!!

[David Ritchie]

I have seen this problem also occur when
I am using pvmove to transfer data disk to
disk online.