vgchange couldn't re-synchronize stale partitions of the logical volume i/o error

[David Freer_1]

Afternoon all.

Got a pesky partition that is refusing to play ball. Following a disk replacement (c6t11d0), one particular partition (in this case /data) is refusing to sync:

# vgsync /dev/vg01
Resynchronized logical volume "/dev/vg01/tools".
Resynchronized logical volume "/dev/vg01/reports".
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgsync: Couldn't resynchronize logical volume "/dev/vg01/data".
vgsync: Couldn't resynchronize volume group "/dev/vg01".

# lvdisplay -v /dev/vg01/data shows that the whole of /dev/vg01/data is stale on c6t11d0 (current on the other disk).

c6t11d0 as a disk is fine as far as I am able to tell (i.e. ioscan, diskinfo, dd, no SCSI errors in syslog etc), it just seems to be this one partition that is having problems. Any suggestions are welcomed...I am faced with a 100 mile trip to reinstall the system otherwise! :(

[Torsten.]

I would first check (wit "dd") if the remaining disk is OK (must not have stale extends), then check the new disk if "dd" can run without errors.

[Tim Nelson]

One of the nuances with LVM mirroring is that the failed disk may not be the one labeled as stale.

Before replacing a mirrored disk always do as Torsten mentioned. read both disks with dd to determine which disk is bad. dd if=/dev/dsk/cxtxdx of=/dev/null

LVM may indicate one as stale because it cold not sync with the other.

[David Freer_1]

Thanks for the prompt response. As mentioned in the text, dd suggests the disk is working fine...

[Tim Nelson]

sorry about missing that part.

I would then assume then that the extent labeled as stale is stale because it could not read from the bad disk.

Obviously now that the bad disk has been replaced you have a hole. Not much you can do other than restore from your latest backup.

where we sure that the bad disk was really bad ?

[Stephen Doud]

Activate the VG manually (vgchange -a y vg01) and do an lvdisplay of of /dev/vg01/data and check the number of "mirror copies" and see if the number of copies matches what you configured.
Example: If you are certain you created a single mirror copy but the number of mirrors copies is 2, the LVM metadata is incorrect.
Consult the HP Support Center for steps to resolve this.

If a defective LVM disk was replaced but the LVM metadata was not restored to the disk, use vgcfgrestore to reload it, then activate the VG again and see if the resync automatically starts.

[David Freer_1]

lvdisplay for /dev/vg01/data shows 1 mirror copy as expected...

[Hakki Aydin Ucar]

I could not find any vgchange trials in your question but vgsync ???

I recommend to use vgchange as Stephen suggested:

vgchange -y a /dev/vg01

if does not work try disable/enable :

vgchange -y n /dev/vg01
vgchange -y a /dev/vg01

[David Freer_1]

Thanks to all who replied but none of these did the trick. Situation is now sorted - details to follow shortly...

[David Freer_1]

Ok a colleague with a lot more experience than I performed the following:

# lvreduce –m 0 /dev/vg01/data
# lvremove –f /dev/vg01/data
# lvcreate –l 1 /dev/vg01/data –m 1 –n data /dev/vg01
# lvextend -L 4000 /dev/vg01/data
# newfs –F vxfs /dev/vg01/rdata

From there we were able to mount /data OK and copy the data over from another system.