Error during resync operation

[chouse]

Getting these errors constantly in syslog:

Mar 12 08:27:45 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 20747648, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608

VG 64 0x050000: LV 7: PVLink 1 0x00000a is always the same, the block to read is different each time. Error number 126 is the same, along with bytes to read and bytes left unread.

The VG referrenced by VG 64 0x050000 is VG status available, with 6 LVs. 5 of the LVs are available/syncd with 1 PV used. The last LV is available/stale with 2 PVs used. The stale LV also has mirror copies set to 1 whereas the other LVs do not have any mirror copies and only occupy one PV.

Looking verbosely at the available/stale LV, we see the vast majority of logical extents (LEs) are on /dev/disk/disk27 and the vast majority of the ones on /dev/disk/disk168 are stale. There is however, one LE that is current on disk168 and stale on disk27, and there are two LVs that are current on both.

00006 /dev/disk/disk27        02182 stale    /dev/disk/disk168       36101 current

00300 /dev/disk/disk27        02476 current  /dev/disk/disk168       36395 current

00301 /dev/disk/disk27        02477 current  /dev/disk/disk168       36396 current

I believe this is left over from a migration to a new set of LUNs in the VG. Would this explain the error messages in syslog, or is something else going on? 

Also, about every 20 minutes we see these for the same LV:

Mar 12 09:00:07 serverName vmunix: LVM: WARNING: VG 64 0x050000: LV 7: Some I/O requests to this LV are waiting
Mar 12 09:00:07 serverName vmunix: indefinitely for an unavailable PV. These requests will be queued until
Mar 12 09:00:07 serverName above message repeats 2 times
Mar 12 09:00:07 serverName vmunix: the PV becomes available (or a timeout is specified for the LV).
Mar 12 09:00:07 serverName vmunix: LVM: NOTICE: VG 64 0x050000: LV 7: All I/O requests to this LV that were
Mar 12 09:00:07 serverName vmunix: waiting indefinitely for an unavailable PV have now completed.
Mar 12 09:00:36 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17839488, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:01:06 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17847680, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:01:06 serverName vmunix: LVM: WARNING: VG 64 0x050000: LV 7: Some I/O requests to this LV are waiting
Mar 12 09:01:06 serverName vmunix: indefinitely for an unavailable PV. These requests will be queued until
Mar 12 09:01:06 serverName vmunix: the PV becomes available (or a timeout is specified for the LV).
Mar 12 09:01:06 serverName vmunix: LVM: NOTICE: VG 64 0x050000: LV 7: All I/O requests to this LV that were
Mar 12 09:01:06 serverName vmunix: waiting indefinitely for an unavailable PV have now completed.

... with more of the "Error during resync operation" messages following:

Mar 12 09:01:35 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17913216, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:02:04 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17921408, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:02:34 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17929600, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:03:03 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17945984, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:03:33 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17962368, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:04:02 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17978752, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:04:31 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 17995136, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:05:01 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 18003328, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:05:30 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 18011520, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:06:00 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 18019712, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:06:29 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 18027904, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608
Mar 12 09:06:58 serverName vmunix: LVM: Error during resync operation VG 64 0x050000: LV 7: PVLink 1 0x00000a Read at block offset: 18044288, error Number 126, number of bytes to Read 8388608, bytes left unread 8388608

[Kris_Knigga]

It looks like one of your disks is circling the drain.

When Good Disks Go Bad

[chouse]

Trouble is they are LUNs from an EVA disk array.

I'm curious about the "PVLink 1 0x00000a" part, how can I figure out which disk/lun/path that relates to?

[Kris_Knigga]

Decoding device names from vmunix errors in syslog

Going by that, I'd say the device is major number 1 minor number 0x00000a, which is strange because it looks (on my machines) that devices with major number 1 are serial devices (/dev/mux2, /dev/tty2p0, etc).

[Patrick Wallek]

Look in the /dev/disk directory and find the 00000a minor number.

ll /dev/disk | grep 00000a