HP-UX rx2660 - IO Type : MPT Domain Validation IO has timed-out.

[atari83]

Dears,

I just noticed an abnormality on one of storges (HP MSA 30) that is connected to a HP rx2660 server via SCSI cables, there were no any amber LED on the storage disks, but strange behaviour.

It's like: first, the 7 disks (from left) blinks green for a second and turn off, and then the 7 disks (from right) do green blink. it happen repeatedly!

HP MSA 30 has no any controller and is configured as JBOD, so I start take a look at the server for finding any useful logs:

dmesg log:

Target ID: 3, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: IO Type : MPT Domain Validation IO has timed-out. Target ID: 3, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: IO Type : MPT Domain Validation IO has timed-out. Target ID: 3, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: IO Type : MPT Domain Validation IO has timed-out. Target ID: 3, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: IO Type : MPT Domain Validation IO has timed-out. Target ID: 3, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00

 SCSI: bp: e000000135006800
dev: 1f034000
cdb: 12 00 00 00 80 00
retry_cnt: 5
residual: 80
status: (400) None -- Incomplete

SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.
SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. Inquiry Command - CDB: 12 00 00 00 40 00
SCSI Ultra320 0/3/1/0/4/1 instance 4: External SCSI bus reset detected. Condition cleared, no intervention required.

ioscan:

# ioscan -fnk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (4030)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
tty 0 0/0/1/0 rmp3f01 CLAIMED INTERFACE PCI class(255,0) (103c1303)
tty 1 0/0/1/1 rmp3f01 CLAIMED INTERFACE PCI SimpleComm (103c1302)
tty 2 0/0/1/2 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/diag/mux2 /dev/mux2 /dev/tty2p0
OO 0 0/0/2/0 UsbOhci CLAIMED INTERFACE USB OHCI Interface
OO 1 0/0/2/0.0 UsbMiniBus CLAIMED INTERFACE USB Composite Device
OO 2 0/0/2/0.0.0 UsbBootKeyboard CLAIMED DEVICE Virtual Keyboard
unknown -1 0/0/2/0.0.1 UNCLAIMED UNKNOWN OO Device Driver
OO 4 0/0/2/0.2 UsbMiniBus CLAIMED INTERFACE USB Composite Device
OO 5 0/0/2/0.2.0 UsbBootKeyboard CLAIMED DEVICE HID Keyboard
OO 7 0/0/2/0.2.1 UsbBootMouse CLAIMED DEVICE USB Boot Protocol Mouse
OO 3 0/0/2/1 UsbOhci CLAIMED INTERFACE USB OHCI Interface
OO 8 0/0/2/1.0 UsbMiniBus CLAIMED INTERFACE USB Composite Device
OO 9 0/0/2/1.0.0 UsbBulkOnlyMS CLAIMED DEVICE USB Bulk Only
ext_bus 0 0/0/2/1.0.16 UsbScsiAdaptor CLAIMED INTERFACE USB SCSI Stack Adaptor
target 0 0/0/2/1.0.16.0 tgt CLAIMED DEVICE
disk 3 0/0/2/1.0.16.0.0 sdisk CLAIMED DEVICE TEAC DVD-ROM DW-224EV
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
target 1 0/0/2/1.0.16.7 tgt CLAIMED DEVICE
ctl 7 0/0/2/1.0.16.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
OO 6 0/0/2/2 UsbEhci CLAIMED INTERFACE USB EHCI Interface
graphics 0 0/0/3/0 gvid_core CLAIMED INTERFACE PCI Display (1002515e)
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (122e)
escsi_ctlr 0 0/1/1/0 sasd CLAIMED INTERFACE HP PCI/PCI-X SAS MPT Adapter
/dev/sasd0
ext_bus 7 0/1/1/0.0.0 sasd_vbus CLAIMED INTERFACE SAS Device Interface
target 33 0/1/1/0.0.0.0 tgt CLAIMED DEVICE
disk 1 0/1/1/0.0.0.0.0 sdisk CLAIMED DEVICE HP DG146BB976
/dev/dsk/c7t0d0 /dev/dsk/c7t0d0s2 /dev/rdsk/c7t0d0 /dev/rdsk/c7t0d0s2
/dev/dsk/c7t0d0s1 /dev/dsk/c7t0d0s3 /dev/rdsk/c7t0d0s1 /dev/rdsk/c7t0d0s3
target 34 0/1/1/0.0.0.1 tgt CLAIMED DEVICE
disk 2 0/1/1/0.0.0.1.0 sdisk CLAIMED DEVICE HP DG146BB976
/dev/dsk/c7t1d0 /dev/dsk/c7t1d0s2 /dev/rdsk/c7t1d0 /dev/rdsk/c7t1d0s2
/dev/dsk/c7t1d0s1 /dev/dsk/c7t1d0s3 /dev/rdsk/c7t1d0s1 /dev/rdsk/c7t1d0s3
lan 0 0/1/2/0 igelan CLAIMED INTERFACE HP PCI-X 1000Base-T Dual-port Built-in
lan 1 0/1/2/1 igelan CLAIMED INTERFACE HP PCI-X 1000Base-T Dual-port Built-in
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (12ee)
ba 3 0/2/1/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 1 0/2/1/0/4/0 mpt CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
/dev/mpt1
target 2 0/2/1/0/4/0.0 tgt CLAIMED DEVICE
disk 25 0/2/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
target 3 0/2/1/0/4/0.1 tgt CLAIMED DEVICE
disk 26 0/2/1/0/4/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
target 4 0/2/1/0/4/0.2 tgt CLAIMED DEVICE
disk 27 0/2/1/0/4/0.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
target 5 0/2/1/0/4/0.3 tgt CLAIMED DEVICE
disk 28 0/2/1/0/4/0.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t3d0 /dev/rdsk/c1t3d0
target 6 0/2/1/0/4/0.4 tgt CLAIMED DEVICE
disk 29 0/2/1/0/4/0.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t4d0 /dev/rdsk/c1t4d0
target 7 0/2/1/0/4/0.5 tgt CLAIMED DEVICE
disk 30 0/2/1/0/4/0.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t5d0 /dev/rdsk/c1t5d0
target 8 0/2/1/0/4/0.6 tgt CLAIMED DEVICE
ctl 8 0/2/1/0/4/0.6.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t6d0
target 9 0/2/1/0/4/0.8 tgt CLAIMED DEVICE
disk 31 0/2/1/0/4/0.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t8d0 /dev/rdsk/c1t8d0
ext_bus 2 0/2/1/0/4/1 mpt CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
/dev/mpt2
target 10 0/2/1/0/4/1.0 tgt CLAIMED DEVICE
disk 4 0/2/1/0/4/1.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
target 11 0/2/1/0/4/1.1 tgt CLAIMED DEVICE
disk 9 0/2/1/0/4/1.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
target 12 0/2/1/0/4/1.2 tgt CLAIMED DEVICE
disk 11 0/2/1/0/4/1.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
target 13 0/2/1/0/4/1.3 tgt CLAIMED DEVICE
disk 14 0/2/1/0/4/1.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t3d0 /dev/rdsk/c2t3d0
target 14 0/2/1/0/4/1.4 tgt CLAIMED DEVICE
disk 17 0/2/1/0/4/1.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t4d0 /dev/rdsk/c2t4d0
target 15 0/2/1/0/4/1.5 tgt CLAIMED DEVICE
disk 20 0/2/1/0/4/1.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t5d0 /dev/rdsk/c2t5d0
target 16 0/2/1/0/4/1.6 tgt CLAIMED DEVICE
ctl 9 0/2/1/0/4/1.6.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t6d0
target 17 0/2/1/0/4/1.8 tgt CLAIMED DEVICE
disk 23 0/2/1/0/4/1.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t8d0 /dev/rdsk/c2t8d0
lan 2 0/2/1/0/6/0 iether CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 3 0/2/1/0/6/1 iether CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ba 4 0/3 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (12ee)
ba 5 0/3/1/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 3 0/3/1/0/4/0 mpt CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
/dev/mpt3
target 18 0/3/1/0/4/0.0 tgt CLAIMED DEVICE
disk 5 0/3/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t0d0 /dev/rdsk/c3t0d0
target 19 0/3/1/0/4/0.1 tgt CLAIMED DEVICE
disk 7 0/3/1/0/4/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t1d0 /dev/rdsk/c3t1d0
target 20 0/3/1/0/4/0.2 tgt CLAIMED DEVICE
disk 12 0/3/1/0/4/0.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
target 21 0/3/1/0/4/0.3 tgt CLAIMED DEVICE
disk 15 0/3/1/0/4/0.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t3d0 /dev/rdsk/c3t3d0
target 35 0/3/1/0/4/0.5 tgt NO_HW DEVICE
disk 21 0/3/1/0/4/0.5.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c3t5d0 /dev/rdsk/c3t5d0
target 36 0/3/1/0/4/0.6 tgt NO_HW DEVICE
ctl 10 0/3/1/0/4/0.6.0 sctl NO_HW DEVICE Initiator
/dev/rscsi/c3t6d0
target 37 0/3/1/0/4/0.8 tgt NO_HW DEVICE
disk 24 0/3/1/0/4/0.8.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c3t8d0 /dev/rdsk/c3t8d0
target 38 0/3/1/0/4/0.15 tgt NO_HW DEVICE
ctl 12 0/3/1/0/4/0.15.0 sctl NO_HW DEVICE COMPAQ PROLIANT 7L7E*DB
ext_bus 4 0/3/1/0/4/1 mpt CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
/dev/mpt4
target 22 0/3/1/0/4/1.0 tgt NO_HW DEVICE
disk 6 0/3/1/0/4/1.0.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
target 23 0/3/1/0/4/1.1 tgt CLAIMED DEVICE
disk 8 0/3/1/0/4/1.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c4t1d0 /dev/rdsk/c4t1d0
target 24 0/3/1/0/4/1.2 tgt CLAIMED DEVICE
disk 10 0/3/1/0/4/1.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c4t2d0 /dev/rdsk/c4t2d0
target 25 0/3/1/0/4/1.3 tgt NO_HW DEVICE
disk 13 0/3/1/0/4/1.3.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
target 26 0/3/1/0/4/1.4 tgt CLAIMED DEVICE
disk 16 0/3/1/0/4/1.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c4t4d0 /dev/rdsk/c4t4d0
target 27 0/3/1/0/4/1.5 tgt CLAIMED DEVICE
disk 19 0/3/1/0/4/1.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c4t5d0 /dev/rdsk/c4t5d0
target 28 0/3/1/0/4/1.6 tgt CLAIMED DEVICE
ctl 11 0/3/1/0/4/1.6.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t6d0
target 29 0/3/1/0/4/1.8 tgt CLAIMED DEVICE
disk 22 0/3/1/0/4/1.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
lan 4 0/3/1/0/6/0 iether CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 5 0/3/1/0/6/1 iether CLAIMED INTERFACE HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ba 6 0/4 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (12ee)
ext_bus 5 0/4/1/0 mpt CLAIMED INTERFACE SCSI Ultra320 A6961-60111
/dev/mpt5
target 31 0/4/1/0.7 tgt CLAIMED DEVICE
ctl 5 0/4/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c5t7d0
ext_bus 6 0/4/1/1 mpt CLAIMED INTERFACE SCSI Ultra320 A6961-60111
/dev/mpt6
target 30 0/4/1/1.2 tgt CLAIMED DEVICE
tape 1 0/4/1/1.2.0 stape CLAIMED DEVICE HP C7438A
/dev/rmt/1m /dev/rmt/1mnb /dev/rmt/c6t2d0BESTn /dev/rmt/c6t2d0DDSb
/dev/rmt/1mb /dev/rmt/c6t2d0BEST /dev/rmt/c6t2d0BESTnb /dev/rmt/c6t2d0DDSn
/dev/rmt/1mn /dev/rmt/c6t2d0BESTb /dev/rmt/c6t2d0DDS /dev/rmt/c6t2d0DDSnb
target 32 0/4/1/1.7 tgt CLAIMED DEVICE
ctl 6 0/4/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c6t7d0
processor 0 120 processor CLAIMED PROCESSOR Processor
processor 1 121 processor CLAIMED PROCESSOR Processor
ba 7 250 pdh CLAIMED BUS_NEXUS Core I/O Adapter
ipmi 0 250/0 ipmi CLAIMED INTERFACE IPMI Controller
/dev/ipmi
tty 3 250/1 asio0 CLAIMED INTERFACE Built-in RS232C
/dev/diag/mux3 /dev/mux3 /dev/tty3p0
acpi_node 0 250/2 acpi_node CLAIMED INTERFACE Acpi Hardware

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
====================================================================================
disk 3 0/0/2/1.0.16.0.0 sdisk CLAIMED DEVICE TEAC DVD-ROM DW-224EV
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0.0.0 sdisk CLAIMED DEVICE HP DG146BB976
/dev/dsk/c7t0d0 /dev/dsk/c7t0d0s2 /dev/rdsk/c7t0d0 /dev/rdsk/c7t0d0s2
/dev/dsk/c7t0d0s1 /dev/dsk/c7t0d0s3 /dev/rdsk/c7t0d0s1 /dev/rdsk/c7t0d0s3
disk 2 0/1/1/0.0.0.1.0 sdisk CLAIMED DEVICE HP DG146BB976
/dev/dsk/c7t1d0 /dev/dsk/c7t1d0s2 /dev/rdsk/c7t1d0 /dev/rdsk/c7t1d0s2
/dev/dsk/c7t1d0s1 /dev/dsk/c7t1d0s3 /dev/rdsk/c7t1d0s1 /dev/rdsk/c7t1d0s3
disk 25 0/2/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 26 0/2/1/0/4/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
disk 27 0/2/1/0/4/0.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 28 0/2/1/0/4/0.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t3d0 /dev/rdsk/c1t3d0
disk 29 0/2/1/0/4/0.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t4d0 /dev/rdsk/c1t4d0
disk 30 0/2/1/0/4/0.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t5d0 /dev/rdsk/c1t5d0
disk 31 0/2/1/0/4/0.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c1t8d0 /dev/rdsk/c1t8d0
disk 4 0/2/1/0/4/1.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 9 0/2/1/0/4/1.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 11 0/2/1/0/4/1.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 14 0/2/1/0/4/1.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t3d0 /dev/rdsk/c2t3d0
disk 17 0/2/1/0/4/1.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t4d0 /dev/rdsk/c2t4d0
disk 20 0/2/1/0/4/1.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t5d0 /dev/rdsk/c2t5d0
disk 23 0/2/1/0/4/1.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c2t8d0 /dev/rdsk/c2t8d0
disk 5 0/3/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t0d0 /dev/rdsk/c3t0d0
disk 7 0/3/1/0/4/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t1d0 /dev/rdsk/c3t1d0
disk 12 0/3/1/0/4/0.2.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 15 0/3/1/0/4/0.3.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t3d0 /dev/rdsk/c3t3d0
disk 21 0/3/1/0/4/0.5.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t5d0 /dev/rdsk/c3t5d0
disk 24 0/3/1/0/4/0.8.0 sdisk CLAIMED DEVICE COMPAQ BF07284961
/dev/dsk/c3t8d0 /dev/rdsk/c3t8d0
disk 6 0/3/1/0/4/1.0.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 8 0/3/1/0/4/1.1.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t1d0 /dev/rdsk/c4t1d0
disk 10 0/3/1/0/4/1.2.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t2d0 /dev/rdsk/c4t2d0
disk 13 0/3/1/0/4/1.3.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
disk 16 0/3/1/0/4/1.4.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t4d0 /dev/rdsk/c4t4d0
disk 19 0/3/1/0/4/1.5.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t5d0 /dev/rdsk/c4t5d0
disk 22 0/3/1/0/4/1.8.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0

print_manifest:

Storage devices HW Path Interface
TEAC DVD-ROM DW-224EV 0/0/2/1.0.16.0.0 USB SCSI Stack Adaptor
HP C7438A 0/4/1/1.2.0 SCSI Ultra320 A6961-60111
COMPAQ BF07284961 0/3/1/0/4/1.8.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.5.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.4.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.3.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.2.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.1.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/1.0.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.8.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.5.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.3.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.2.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.1.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/3/1/0/4/0.0.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.8.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.5.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.4.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.3.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.2.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.1.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/1.0.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.8.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.5.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.4.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.3.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.2.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.1.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
COMPAQ BF07284961 0/2/1/0/4/0.0.0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
HP DG146BB976 140014 Mb 0/1/1/0.0.0.1.0 SAS Device Interface
HP DG146BB976 140014 Mb 0/1/1/0.0.0.0.0 SAS Device Interface

I/O Interfaces
Class H/W Path Driver Description
tty 0/0/1/0 rmp3f01 PCI class(255,0) (103c1303)
tty 0/0/1/1 rmp3f01 PCI SimpleComm (103c1302)
tty 0/0/1/2 asio0 PCI Serial (103c1048)
OO 0/0/2/0 UsbOhci USB OHCI Interface
OO 0/0/2/0.0 UsbMiniBus USB Composite Device
OO 0/0/2/0.2 UsbMiniBus USB Composite Device
OO 0/0/2/1 UsbOhci USB OHCI Interface
OO 0/0/2/1.0 UsbMiniBus USB Composite Device
ext_bus 0/0/2/1.0.16 UsbScsiAdaptor USB SCSI Stack Adaptor
OO 0/0/2/2 UsbEhci USB EHCI Interface
graphics 0/0/3/0 gvid_core PCI Display (1002515e)
escsi_ctlr 0/1/1/0 sasd HP PCI/PCI-X SAS MPT Adapter
ext_bus 0/1/1/0.0.0 sasd_vbus SAS Device Interface
lan 0/1/2/0 igelan HP PCI-X 1000Base-T Dual-port Built-in
lan 0/1/2/1 igelan HP PCI-X 1000Base-T Dual-port Built-in
ext_bus 0/2/1/0/4/0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ext_bus 0/2/1/0/4/1 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 0/2/1/0/6/0 r HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 0/2/1/0/6/1 r HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ext_bus 0/3/1/0/4/0 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ext_bus 0/3/1/0/4/1 HP AB290-60001 PCI/PCI-X U320 SCSI 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 0/3/1/0/6/0 r HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan 0/3/1/0/6/1 r HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
ext_bus 0/4/1/0 mpt SCSI Ultra320 A6961-60111
ext_bus 0/4/1/1 mpt SCSI Ultra320 A6961-60111
ipmi 250/0 ipmi IPMI Controller
tty 250/1 asio0 Built-in RS232C
acpi_node 250/2 acpi_node Acpi Hardware

Could anyone help me to troubleshoot the problem ? Is it disk faiure on storage or could it be SCSI card failure ?

Thanks.

[Robert_Jewell]

The blinking LEDs on the disks are the result of the SCSI controller resets that are happening.

I see that the MSA30 is connected to both ports of a single adapter card wtih two ports.  I dont know how your storage is being utilized, but this configuration can make issues like this more difficult to troubleshoot and you lose redudancy if the card fails; which is what I think is happening...

The controller in the server is failing causing the SCSI timeouts and NO_HW entries on both sides of the MSA.  Even the SCSI Initiator is reporting as NO_HW:

ctl 10 0/3/1/0/4/0.6.0 sctl NO_HW DEVICE Initiator

My thoughts?  Try to reseat or replace that card and check the SCSI cabling and termination.

-Bob

[atari83]

Robert, thanks alot for your reply, and sorry for my delay in sending reply.

Actually, in mean time I tried to get into more details.

So, here it is:

There are two servers (rx2660) which are connected to two MSA30 storages.

These servers are configured as HA cluster, the one I already published the logs is known as server-2 and currently is Active node.

Each server has two SCSI cards, each SCSI board has two SCSI ports.

Each SCSI board is connected too one storage, so deployment is like this:

server-1 , SCSI board 1, SCSI port 1 ---> Storage 1, SCSI module, port 1

server-1 , SCSI board 1, SCSI port 2 ---> Storage 1, SCSI module, port 2

server-2 , SCSI board 1, SCSI port 1 ---> Storage 1, SCSI module, port 3

server-2 , SCSI board 1, SCSI port 2 ---> Storage 1, SCSI module, port 4

server-1 , SCSI board 2, SCSI port 1 ---> Storage 2, SCSI module, port 1

server-1 , SCSI board 2, SCSI port 2 ---> Storage 2, SCSI module, port 2

server-2 , SCSI board 2, SCSI port 1 ---> Storage 2, SCSI module, port 3

server-2 , SCSI board 2, SCSI port 2 ---> Storage 2, SCSI module, port 4

 The problem that we have observed is happening on storage 1. (abnormal hard disk green blinking)

I've already took the log to check how server-1 is responding to storage 1 (as we saw before on server 2, there lots of NO_HW on storage 1 paths)

It was intresting that server-1  respond almost the same ... for making it more clear I have attached both servers logs here.

server-1: http://pastebin.com/8i3VS8ks

server-2: http://pastebin.com/jMLZTtcp

Based on this information , could we summerize that storage 1 SCSI module has some problem ?

 ps: seems i could only attach picture to the post,

Thanks.

[Robert_Jewell]

That very could be.  From the errors you have shared it seems the following are consistent:

SCSI Ultra320 0/3/1/0/4/0 instance 3: IO Type : MPT Domain Validation IO has timed-out. Target ID: 4, LUN ID: 0. 
SCSI Ultra320 0/3/1/0/4/1 instance 4: IO Type : MPT Domain Validation IO has timed-out. Target ID: 3, LUN ID: 0. 

Server2 shows that this target 3 is not being seen while others on that path are.

1. target      25  0/3/1/0/4/1.3   tgt             NO_HW       DEVICE      
2. disk        13  0/3/1/0/4/1.3.0   sdisk           NO_HW       DEVICE       COMPAQ  BF07284961

There are nother NO_HW instances, but I think I would start there and try removing/repalcing that disk module and see if that improves anything.

-Bob

[atari83]

Hey Bob,

Yes, didnt notice that :)

So, what do you think ? can i just unplug and plug again the same disk in normal operational time, or do i need to do it in down-time ?

[atari83]

just one thing that makes me confuse ..

in ioscan -fnk as you pointed out that path was NO_HW, but in same log file, and ioscan -fnC disk the path is CLAIMED!

disk        14  0/2/1/0/4/1.3.0   sdisk          CLAIMED     DEVICE       COMPAQ  BF07284961

                               /dev/dsk/c2t3d0   /dev/rdsk/c2t3d0

edited: my bad ... different path :)

[Robert_Jewell]

That I cannot answer.  It depends on how your volumes are configured, how the storage is shared, etc.  If you are not sure, then its alway safer doing such work during a maintence window.

-Bob

[atari83]

still confused :|

ioscan -fnk:

target 25 0/3/1/0/4/1.3 tgt NO_HW DEVICE
disk 13 0/3/1/0/4/1.3.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
target 26 0/3/1/0/4/1.4 tgt CLAIMED DEVICE
disk 16 0/3/1/0/4/1.4.0 sdisk CLAIMED DEVICE COMPAQ BF07284961

 ioscan -fnC disk:

disk 13 0/3/1/0/4/1.3.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t3d0 /dev/rdsk/c4t3d0
disk 16 0/3/1/0/4/1.4.0 sdisk NO_HW DEVICE COMPAQ BF07284961
/dev/dsk/c4t4d0 /dev/rdsk/c4t4d0

[atari83]

it is intresting that the results of ioscan -fnk and ioscan -fnC disk doesnt match ..

Could it be cause of constant reseting the SCSI path that you mentioned earlier ? I mean the time gap between when i issued the ioscan with -fnk and ioscan with -fnC disk ?

[Robert_Jewell]

When you specify the "k" option, you are not scanning the busses; its reading the data from memory.  Leave off the "-k" and the hardware is scanned for devices.  This takes longer to run.

Also, when you specify the "-C disk" option, you are filtering by the given class, in this case "disk".  Everything else is ommitted.

In counting each instance of a disk in both outputs, I see that there are 30 disks shown.

-Bob