Operating System - Linux
1748170 Members
4324 Online
108758 Solutions
New Discussion

Re: kernel: iSCSI: session f7132000 failing itt 407856161 task cdcc3164 cmnd eff5d800 cdb 0x2a to (2 0

 
skt_skt
Honored Contributor

kernel: iSCSI: session f7132000 failing itt 407856161 task cdcc3164 cmnd eff5d800 cdb 0x2a to (2 0


Red Hat Enterprise Linux AS release 3 (Taroon Update 7)

iscsi-initiator-utils-3.6.3-3

file systems bacame read only mode as a few LUNS reported not ready in messages.
LUNs(4),( LUN names: lun0/lun1/lun2/lun3/l) were unable to write; unmounted and mounted the file systems and it brought the luns back to write and read normal status.

here is the status of the LUNS effected.

/vol/proddb/tacla138p/lun0 (393 days, 4 hours, 44 minutes, 1 second)
Read (kbytes) Write (kbytes) Read Ops Write Ops Other Ops QFulls Partner Ops Partner KBytes
422229100 1646007101 10723023 161737727 121 0 0 0

/vol/proddb/tacla138p/lun1 (393 days, 4 hours, 44 minutes, 1 second)
Read (kbytes) Write (kbytes) Read Ops Write Ops Other Ops QFulls Partner Ops Partner KBytes
155104864 1512298525 8909577 117869883 118 0 0 0

vol/proddb/tacla138p/lun2 (393 days, 4 hours, 44 minutes, 1 second)
Read (kbytes) Write (kbytes) Read Ops Write Ops Other Ops QFulls Partner Ops Partner KBytes
5434485 459571036 433096 20476576 118 0 0 0

/vol/proddb/tacla138p/lun3 (393 days, 4 hours, 44 minutes, 1 second)
Read (kbytes) Write (kbytes) Read Ops Write Ops Other Ops QFulls Partner Ops Partner KBytes
22328237 59885088 387978 1424247 118 0 0 0

Following messages appeared on the host

Jun 1 13:19:53 tacld138p sshd(pam_unix)[22820]: session closed for user root
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856161 task cdcc3164 cmnd eff5d800 cdb 0x2a to (2 0 0 0) at 4294937836, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: Device 08:20 not ready.
Jun 1 13:26:52 tacld138p kernel: I/O error: dev 08:20, sector 71685864
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 recv_cmd itt 407856161, task cdcc3164, refcount 0, no SCSI command at 4294937836
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 already completed itt 407856161, task cdcc3164, (2 0 0 0)
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856165 task cdcc3164 cmnd ef8d4c00 cdb 0x28 to (2 0 0 3) at 4294937842, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: Device 08:50 not ready.
Jun 1 13:26:52 tacld138p kernel: I/O error: dev 08:50, sector 2298224
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856166 task f5cd3808 cmnd ef8d4c00 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856167 task f17146dc cmnd efd96c00 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856168 task e86bf678 cmnd efd96800 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856169 task ef042d80 cmnd ef8d4e00 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856170 task f1710420 cmnd efd96200 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856171 task f1710484 cmnd ef8d4600 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: iSCSI: session f7132000 failing itt 407856172 task cdccb1c8 cmnd efd96600 cdb 0x28 to (2 0 0 3) at 4294937843, retries 0, allowed 5
Jun 1 13:26:52 tacld138p kernel: Device 08:50 not ready.
Jun 1 13:26:52 tacld138p kernel: I/O error: dev 08:50, sector 277712
Jun 1 13:26:52 tacld138p kernel: Device 08:50 not ready.
Jun 1 13:26:52 tacld138p kernel: I/O error: dev 08:50, sector 294288
Jun 1 13:26:52 tacld138p kernel: Device 08:50 not ready.

Any one familier with this errors? As i can see ISCSI re-connect (a result of the unmount/mount). But, I don't see were it disconnected
1 REPLY 1
Steven E. Protter
Exalted Contributor

Re: kernel: iSCSI: session f7132000 failing itt 407856161 task cdcc3164 cmnd eff5d800 cdb 0x2a to (2 0

Shalom,

Based on experience, there is a SAN problem. It may be cables, switch or the array itself. It may be the system is pushing I/O limits too hard.

This shows an intermittent failure that is somewhere in the SAN. IMHO.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com