Unable to Remove Stale Extents After Mirrored

[itrc55]

Hi,

i facing this error message when i try to reduce filesystem from old disk

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

 #lvreduce -m 0  /dev/oravg/lv_olnk /dev/disk/disk7 /dev/disk/disk17 /dev/disk/disk18 /dev/disk/disk19
Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/oravg/lv_olnk".

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

i mirrored this logical volume with new diskas as following

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

lvextend -m 1 /dev/oravg/lv_olnk /dev/disk/disk39 /dev/disk/disk40 /dev/disk/disk41 /dev/disk/disk42

then i see this message

lvextend: Couldn't resynchronize logical volume "/dev/oravg/lv_olnk".
The newly allocated mirrors are now being synchronized. This operation will
take some time. Please wait ....
lvextend: Couldn't resynchronize stale partitions of the logical volume:
Device offline/Powerfailed

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

when i chek the filesystem mirroring status i see it's mirrored

>>>>>>>>>>>>>>>>>>>>>>>>>

 #lvdisplay /dev/oravg/lv_olnk
--- Logical volumes ---
LV Name                     /dev/oravg/lv_olnk
VG Name                     /dev/oravg
LV Permission               read/write
LV Status                   available/stale
Mirror copies               1
Consistency Recovery        MWC
Schedule                    parallel
LV Size (Mbytes)            51200
Current LE                  800
Allocated PE                1600
Stripes                     0
Stripe Size (Kbytes)        0
Bad block                   NONE
Allocation                  strict
IO Timeout (Seconds)        default
Number of Snapshots         0
<<<<<<<<<<<<<<<<<<<<<<<<<

 #lvdisplay -v /dev/oravg/lv_olnk | grep stale
LV Status                   available/stale
   00000000 /dev/disk/disk7         00001584 stale    /dev/disk/disk39        00001584 current
   00000030 /dev/disk/disk17        00000015 current  /dev/disk/disk40        00000015 stale
   00000031 /dev/disk/disk17        00000016 current  /dev/disk/disk40        00000016 stale
   00000038 /dev/disk/disk17        00000023 current  /dev/disk/disk40        00000023 stale
   00000047 /dev/disk/disk17        00000032 current  /dev/disk/disk40        00000032 stale
   00000048 /dev/disk/disk17        00000033 stale    /dev/disk/disk40        00000033 current
   00000049 /dev/disk/disk17        00000034 current  /dev/disk/disk40        00000034 stale
   00000051 /dev/disk/disk17        00000036 stale    /dev/disk/disk40        00000036 current
   00000054 /dev/disk/disk17        00000039 stale    /dev/disk/disk40        00000039 current
   00000055 /dev/disk/disk17        00000040 stale    /dev/disk/disk40        00000040 current
   00000056 /dev/disk/disk17        00000041 stale    /dev/disk/disk40        00000041 current
   00000059 /dev/disk/disk17        00000044 current  /dev/disk/disk40        00000044 stale
   00000064 /dev/disk/disk17        00000049 stale    /dev/disk/disk40        00000049 current
   00000065 /dev/disk/disk17        00000050 stale    /dev/disk/disk40        00000050 current
   00000068 /dev/disk/disk17        00000053 stale    /dev/disk/disk40        00000053 current
   00000077 /dev/disk/disk17        00000062 current  /dev/disk/disk40        00000062 stale
   00000081 /dev/disk/disk17        00000066 stale    /dev/disk/disk40        00000066 current
   00000085 /dev/disk/disk17        00000070 current  /dev/disk/disk40        00000070 stale
   00000087 /dev/disk/disk17        00000072 current  /dev/disk/disk40        00000072 stale
   00000097 /dev/disk/disk17        00000082 stale    /dev/disk/disk40        00000082 current
   00000098 /dev/disk/disk17        00000083 stale    /dev/disk/disk40        00000083 current
   00000099 /dev/disk/disk17        00000084 current  /dev/disk/disk40        00000084 stale
   00000105 /dev/disk/disk17        00000090 current  /dev/disk/disk40        00000090 stale
   00000111 /dev/disk/disk17        00000096 current  /dev/disk/disk40        00000096 stale
   00000115 /dev/disk/disk17        00000100 current  /dev/disk/disk40        00000100 stale
   00000116 /dev/disk/disk17        00000101 current  /dev/disk/disk40        00000101 stale
   00000117 /dev/disk/disk17        00000102 stale    /dev/disk/disk40        00000102 current
   00000122 /dev/disk/disk17        00000107 current  /dev/disk/disk40        00000107 stale
   00000123 /dev/disk/disk17        00000108 current  /dev/disk/disk40        00000108 stale
   00000126 /dev/disk/disk17        00000111 current  /dev/disk/disk40        00000111 stale
   00000127 /dev/disk/disk17        00000112 current  /dev/disk/disk40        00000112 stale
   00000128 /dev/disk/disk17        00000113 current  /dev/disk/disk40        00000113 stale
   00000131 /dev/disk/disk17        00000116 current  /dev/disk/disk40        00000116 stale
   00000147 /dev/disk/disk17        00000132 current  /dev/disk/disk40        00000132 stale
   00000148 /dev/disk/disk17        00000133 current  /dev/disk/disk40        00000133 stale
   00000149 /dev/disk/disk17        00000134 current  /dev/disk/disk40        00000134 stale
   00000162 /dev/disk/disk17        00000147 current  /dev/disk/disk40        00000147 stale
   00000167 /dev/disk/disk17        00000152 current  /dev/disk/disk40        00000152 stale
   00000169 /dev/disk/disk17        00000154 current  /dev/disk/disk40        00000154 stale
   00000176 /dev/disk/disk17        00000161 current  /dev/disk/disk40        00000161 stale
   00000194 /dev/disk/disk17        00000179 current  /dev/disk/disk40        00000179 stale
   00000195 /dev/disk/disk17        00000180 current  /dev/disk/disk40        00000180 stale
   00000196 /dev/disk/disk17        00000181 current  /dev/disk/disk40        00000181 stale
   00000201 /dev/disk/disk17        00000186 current  /dev/disk/disk40        00000186 stale
   00000202 /dev/disk/disk17        00000187 current  /dev/disk/disk40        00000187 stale
   00000203 /dev/disk/disk17        00000188 current  /dev/disk/disk40        00000188 stale
   00000256 /dev/disk/disk17        00000241 current  /dev/disk/disk40        00000241 stale
   00000257 /dev/disk/disk17        00000242 current  /dev/disk/disk40        00000242 stale
   00000258 /dev/disk/disk17        00000243 current  /dev/disk/disk40        00000243 stale
   00000294 /dev/disk/disk17        00000279 current  /dev/disk/disk40        00000279 stale
   00000295 /dev/disk/disk17        00000280 current  /dev/disk/disk40        00000280 stale
   00000297 /dev/disk/disk17        00000282 current  /dev/disk/disk40        00000282 stale
   00000298 /dev/disk/disk17        00000283 current  /dev/disk/disk40        00000283 stale
   00000316 /dev/disk/disk17        00000301 current  /dev/disk/disk40        00000301 stale
   00000320 /dev/disk/disk17        00000305 current  /dev/disk/disk40        00000305 stale
   00000322 /dev/disk/disk17        00000307 current  /dev/disk/disk40        00000307 stale
   00000360 /dev/disk/disk17        00000345 current  /dev/disk/disk40        00000345 stale
   00000361 /dev/disk/disk17        00000346 current  /dev/disk/disk40        00000346 stale
   00000362 /dev/disk/disk17        00000347 current  /dev/disk/disk40        00000347 stale
   00000377 /dev/disk/disk17        00000362 current  /dev/disk/disk40        00000362 stale
   00000379 /dev/disk/disk17        00000364 current  /dev/disk/disk40        00000364 stale
   00000380 /dev/disk/disk17        00000365 current  /dev/disk/disk40        00000365 stale
   00000381 /dev/disk/disk17        00000366 current  /dev/disk/disk40        00000366 stale
   00000388 /dev/disk/disk17        00000373 current  /dev/disk/disk40        00000373 stale
   00000389 /dev/disk/disk17        00000374 current  /dev/disk/disk40        00000374 stale
   00000390 /dev/disk/disk17        00000375 current  /dev/disk/disk40        00000375 stale
   00000422 /dev/disk/disk17        00000407 current  /dev/disk/disk40        00000407 stale
   00000428 /dev/disk/disk17        00000413 current  /dev/disk/disk40        00000413 stale
   00000429 /dev/disk/disk17        00000414 current  /dev/disk/disk40        00000414 stale
   00000584 /dev/disk/disk17        00000569 current  /dev/disk/disk40        00000569 stale
   00000585 /dev/disk/disk17        00000570 current  /dev/disk/disk40        00000570 stale
   00000600 /dev/disk/disk17        00000585 current  /dev/disk/disk40        00000585 stale

please your support, the new disk is disk40 and old disk is disk17

Thanks

[Matti_Kurkela]

#lvreduce -m 0  /dev/oravg/lv_olnk /dev/disk/disk7 /dev/disk/disk17 /dev/disk/disk18 /dev/disk/disk19
Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/oravg/lv_olnk".

You tried to lvreduce 4 PVs from the LV? How many PVs were used by this LV before that?

If the LV was located exclusively on these four PVs, then this message could be expected: lvreduce could not make these PVs unused by this LV. But if that was true, after this operation there should be only one copy of each extent, so your mirroring is now gone. If the disk that holds the only remaining copy of a given extent were to fail, you would lose data...

lvextend: Couldn't resynchronize stale partitions of the logical volume:
Device offline/Powerfailed

...and, unfortunately, this seems to be exactly what happened. 

"Device offline/Powerfailed" means the disk went to NO_HW state on you. It might have been temporary, if the disk hardware just reset itself. Or the disk may have completely died.

When an old disk is starting to fail, you should remember that stressing it may cause it to get worse. Running a mirroring operation (lvextend -m) means reading all data (of that particular LV) from the disk(s) in order to copy it to other disk(s), so it can be a very stressful operation for old disks. Therefore, to minimize the risk of losing data when migrating it off the old disks, you should have added the new mirrors before removing the old ones (lvextend -m 2, then reduce the old mirrors to go back to -m 1).  This way, if one of the old mirrors fails before the the data is copied to the new mirror, the other old mirror can be used to provide the missing data.

When you look at the list of extents in the "lvdisplay -v" output, "current" means "here is an up-to-date version of this extent's data". If the extent is "current" it does not necessarily mean it's readable.

"Stale", on the other hand, might mean that the physical extent has failed and can no longer reliably hold data... or in the case of mirroring, that no data has been successfully copied into that extent yet (because the disk that held the only up-to-date copy failed before the mirroring could be completed).

If you have a good backup of your data, the quickest way out of this mess would probably be to identify the bad disks by checking ioscan and/or dmesg, then unmount and lvremove this LV, re-create it on known-good disks, and restore from backup.

If there is no backup and the ioscan command still shows the disk as CLAIMED, you can try running "lvsync" on the LV a few times to see if the data can be recovered from the old disk by trying to read it over and over. (This is not guaranteed to work: if I were religious, I would also recommend praying at this point.)