syslog disk errors

[Philip Priest]

Jun 4 00:08:43 volapps1 vmunix:
Jun 4 00:08:43 volapps1 vmunix: SCSI: Write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 131342992, sectno: 262685984, offset: 1351237632, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: blkno: 56890056, sectno: 113780112, offset: 2420842496, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: blkno: 46485736, sectno: 92971472, offset: 356753408, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 4096,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 above message repeats 2 times
Jun 4 00:08:43 volapps1 vmunix: blkno: 44622624, sectno: 89245248, offset: 2743894016, bcount: 4096.
Jun 4 00:08:43 volapps1 vmunix: blkno: 44622632, sectno: 89245264, offset: 2743902208, bcount: 4096.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Read error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 4096,
Jun 4 00:08:43 volapps1 vmunix: blkno: 32172696, sectno: 64345392, offset: 2880069632, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 4096,
Jun 4 00:08:43 volapps1 vmunix: blkno: 27004064, sectno: 54008128, offset: 1882357760, bcount: 4096.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 7720, sectno: 15440, offset: 7905280, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Read error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 15640, sectno: 31280, offset: 16015360, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 10344, sectno: 20688, offset: 10592256, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: blkno: 8712, sectno: 17424, offset: 8921088, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861632, sectno: 225723264, offset: 3901161472, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 above message repeats 2 times
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868992, sectno: 225737984, offset: 3908698112, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868960, sectno: 225737920, offset: 3908665344, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868928, sectno: 225737856, offset: 3908632576, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 above message repeats 2 times
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868864, sectno: 225737728, offset: 3908567040, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868800, sectno: 225737600, offset: 3908501504, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 49152,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868752, sectno: 225737504, offset: 3908452352, bcount: 49152.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 16384,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868736, sectno: 225737472, offset: 3908435968, bcount: 16384.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868672, sectno: 225737344, offset: 3908370432, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868608, sectno: 225737216, offset: 3908304896, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868544, sectno: 225737088, offset: 3908239360, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868480, sectno: 225736960, offset: 3908173824, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868416, sectno: 225736832, offset: 3908108288, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868352, sectno: 225736704, offset: 3908042752, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:43 volapps1 above message repeats 5 times
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868320, sectno: 225736640, offset: 3908009984, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868288, sectno: 225736576, offset: 3907977216, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868224, sectno: 225736448, offset: 3907911680, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868160, sectno: 225736320, offset: 3907846144, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868096, sectno: 225736192, offset: 3907780608, bcount: 65536.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 16384,
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 65536,
Jun 4 00:08:44 volapps1 above message repeats 2 times
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868080, sectno: 225736160, offset: 3907764224, bcount: 16384.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 49152,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868032, sectno: 225736064, offset: 3907715072, bcount: 49152.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112868000, sectno: 225736000, offset: 3907682304, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112862080, sectno: 225724160, offset: 3901620224, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112862024, sectno: 225724048, offset: 3901562880, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112862016, sectno: 225724032, offset: 3901554688, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 24576,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861992, sectno: 225723984, offset: 3901530112, bcount: 24576.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 32768,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861960, sectno: 225723920, offset: 3901497344, bcount: 32768.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861952, sectno: 225723904, offset: 3901489152, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861896, sectno: 225723792, offset: 3901431808, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861888, sectno: 225723776, offset: 3901423616, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861832, sectno: 225723664, offset: 3901366272, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861824, sectno: 225723648, offset: 3901358080, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861768, sectno: 225723536, offset: 3901300736, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861760, sectno: 225723520, offset: 3901292544, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861704, sectno: 225723408, offset: 3901235200, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 8192,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861696, sectno: 225723392, offset: 3901227008, bcount: 8192.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Async write error -- dev: b 31 0x180600, errno: 126, resid: 57344,
Jun 4 00:08:43 volapps1 vmunix: blkno: 112861640, sectno: 225723280, offset: 3901169664, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Read error -- dev: b 31 0x180600, errno: 126, resid: 1024,
Jun 4 00:08:43 volapps1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jun 4 00:08:51 volapps1 vmunix: LVM: Performed a switch for Lun ID = 6 (pv = 0x0000000098bfc000), from raw device 0x1f180600 (with pr
iority: 0, and current flags: 0x40) to raw device 0x1f160600 (with priority: 1, and current flags: 0x0).
Jun 4 00:08:43 volapps1 vmunix:
Jun 4 00:08:51 volapps1 above message repeats 48 times
Jun 4 00:08:51 volapps1 vmunix: LVM: VG 64 0x050000: PVLink 31 0x180600 Failed! The PV is still accessible.
Jun 4 00:09:27 volapps1 vmunix: LVM: VG 64 0x050000: PVLink 31 0x180600 Recovered.
Jun 4 00:09:29 volapps1 vmunix: LVM: Performed a switch for Lun ID = 6 (pv = 0x0000000098bfc000), from raw device 0x1f160600 (with pr
iority: 1, and current flags: 0x0) to raw device 0x1f180600 (with priority: 0, and current flags: 0x0).

Does this indicate im having disk issues?

thanks
Phil

[DCE]

Yes - you definitely have a disk issue

See the following thread for further info

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=962008

[A. Clay Stephenson]

Device 0x1f160600
decodes as follows
1F (Block device 31 - disk)
16h - c22
0 - t0
6 - d6

This means that a problem was detected on /dev/dsk/c22t0d6 and because I see d6 this means that this is an array device as opposed to a simple disk. I also note that a transfer to an alternate link was done. This means that an error was detected on the primary path (bad HBA, cable, disk array controller, or terminator) and a switch to the alternate was initiated.

One of the things that you should check is if the timeout is set to the default (30 seconds) which is fine for simple disks but array LUN's should be set to something in the 120-180 second range. This value can be checked using pvdisplay and changed via pvchange.

[skt_skt]

Looks like a path switching happend.Check at the storage end or any possible connectivity issue going to that path

[MHudec]

Jun 4 00:08:43 volapps1 vmunix: blkno: 112861640, sectno: 225723280, offset: 3901169664, bcount: 57344.
Jun 4 00:08:43 volapps1 vmunix: SCSI: Read error -- dev: b 31 0x180600, errno: 126, resid: 1024,
Jun 4 00:08:51 volapps1 vmunix: LVM: Performed a switch for Lun ID = 6 (pv = 0x0000000098bfc000), from raw device 0x1f180600 (with pr
iority: 0, and current flags: 0x40) to raw device 0x1f160600 (with priority: 1, and current flags: 0x0).
Jun 4 00:09:29 volapps1 vmunix: LVM: Performed a switch for Lun ID = 6 (pv = 0x0000000098bfc000), from raw device 0x1f160600 (with pr
iority: 1, and current flags: 0x0) to raw device 0x1f180600 (with priority: 0, and current flags: 0x0).

According to this, issue happened while accessing disk c24t6d0 (1f 180600), and LVM switched to another link to this drive as c22t6d0 (1f 160600). This link switch lasted for around half a minute, and then it switched back.
Did something happened in SAN? Cabling, HBA, switch issues?