Weird error when trying to mirror

[Matt Hearn]

Hey all, when trying to mirror a logical volume (which is currently activated and mounted), I get the following error:

LVM: Unable to map all the LEs to unique PEs.
lvextend: The LVM device driver could not extend the logical volume "/dev/vggamsp1/lvamtdp441".

Any idea what it's talking about? I've never seen anything like it. I was able to mirror the other volumes in the VG without any problems.

Thanks!

[Matt Hearn]

I think I figured it out; it appears to be related to not having enough free extents in a PVG to completely mirror the disk. (Or at least, if I turned off PVG-strict, it mirrors up fine.)

Weird error to get, though. Thanks!

[Henk Geurts]

hi Matt.
looks like lvm problem, so do a
vgdisplay -v vggamsp1
and an
lvdisplay -v /dev/vggamsp1/lvamtdp441
and
an pvdisplay -v for disks in the vggampsp1
to check for inconsistencies ....
or post the output here....

[TwoProc]

The destination drives is probably not big enough(or doesn't have enough free space) to mirror all the extents from the source drive.

You should be able to mirror to more than one destination drive however. So, if you've got another drive in the vg, you can add that to the end of the lvextend command beyond the first designated destination mirror target pv.

[Matt Hearn]

Now it's doing something weird. When I try to mirror, I get I/O errors:

lvsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvsync: Couldn't resynchronize logical volume "/dev/vggamsp1/lvamtdp441".

This also seems to coincide with our customer noticing I/O problems with files in a volume that uses these disks that.

We've replaced several of these disks (which are in a fibre-attached array) multiple times and we're still getting I/O issues! Could there be some kind of weird fibre termination issues? I took a look at PHKL_33066, but it doesn't appear to have a specific symptom that relates to my issue.

Thanks!

[TwoProc]

I think you've got bad extents in the source pv(s).

Use lvdisplay -v /dev/vggamsp1/lvamtdp441 | grep -i stale

to see if there is anything in there that is bad.

Also, from the list of pv's (from lvdisplay -v) run pvdisplay -v /dev/dsk/cxtxdx and grep that output for "stale" - or just look through the output.

[Kent Ostby]

Check also your LVM cumulative patching.

There were some fixes put in for something like this starting with PHKL_30697 and later.

[Matt Hearn]

Here's the status we have now:

We've tried to mirror /dev/vggamsp1/lvamtdp441 with several different physical disks, and we get I/O errors with all of them, along with 2 stale extents on the mirror when we're done. So we're now thinking that something has gotten corrupted on the primary disk, something that LVM can't detect until it gets the I/O error trying to sync a screwy extent.

I tried a pvmove to get all the extents off the suspect primary disk, and got a similar I/O error, except that it doesn't just skip the bad extent and continue (like it does with mirroring), it just stops.

If I try and reduce it down to one mirror and remove the suspect primary, it tells me that the mirror has stale extents, so it won't do it. Which I guess is pretty smart of LVM.

I thought of trying to add a SECOND mirror to the FS, but I don't have enough free extents, and I'm not willing to break the other mirrors to free some up.

Right now I'm trying to come up with options, and the only thing I can think of at the moment is to:

1) Create a new filesystem with all the space available, making sure not to use any free extents on the funny primary disk, then mount it.
2) Copy all the files from the bad LV to the new one.
3) Wipe out the bad LV. (Or umount it and see if maybe an fsck fixes it.)
4) Remove the suspect disk from the volume group.
5) Extend to the new FS, mirror it, mount it in the old location.

What I'm curious about is whether or not filesystem corruption might get copied over in the files. I don't believe it would, because I think the problem is at or underneath the LVM level, but I'm not sure.

My other option is to wipe it all out and arrange for a ~40GB restore. I'm pretty sure the customer isn't ready to try that.