BL35p disconected from MSA1000

[Edgard Matamala]

Hi all,
I have a BL35p with RedHat 3 connected to an MSA1000 with other 5 servers more.

The problem is that randomly the server loses connection with the MSA, this is only reset when restarting the server. After that the server returns to access your hard drive by the MSA.

This server has boot from SAN, Curiously before Beginning with this issue repeated messages are seen as still in the log messages:


==============================================
Dec 16 16:20:46 isis kernel: qla2x00: ISP System Error - mbx1=1c36h, mbx2=2h, mbx3=2h.<6>qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:20:49 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:20:49 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:20:49 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:20:49 isis kernel: scsi(0): LIP occurred.
Dec 16 16:20:49 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:20:49 isis kernel: scsi(0): Port database changed.
Dec 16 16:20:49 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:20:49 isis kernel: scsi(0): Port database changed.
Dec 16 16:20:49 isis last message repeated 2 times
Dec 16 16:20:55 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:20:55 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:29:00 isis kernel: qla2x00_ioctl_error_recovery(0) issuing device reset
Dec 16 16:29:00 isis kernel: qla2xxx_eh_device_reset(): **** CMD derives a NULL TGT_Q
Dec 16 16:29:00 isis kernel: qla2x00_ioctl_error_recovery(0) elevation to host_reset
Dec 16 16:29:00 isis kernel: scsi(0:0:255:0): now issue ADAPTER RESET.
Dec 16 16:29:00 isis kernel: qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:29:02 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:29:02 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:29:02 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:29:02 isis kernel: scsi(0): LIP occurred.
Dec 16 16:29:02 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:29:02 isis kernel: scsi(0): Port database changed.
Dec 16 16:29:02 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:29:02 isis kernel: scsi(0): Port database changed.
Dec 16 16:29:09 isis last message repeated 2 times
Dec 16 16:29:09 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:29:09 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:29:09 isis kernel: qla2xxx_eh_host_reset(0): reset succeded
Dec 16 16:29:09 isis kernel: qla2x00_ioctl_error_recovery(0) return_status=2002
Dec 16 16:29:09 isis kernel: qla2xxx_eh_abort: cmd already done sp=00000000
Dec 16 16:29:09 isis last message repeated 10 times
Dec 16 16:39:16 isis kernel: qla2x00: Status Entry invalid handle.
Dec 16 16:39:16 isis kernel: qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:39:19 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:39:19 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:39:19 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:39:19 isis kernel: scsi(0): LIP occurred.
Dec 16 16:39:19 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:39:19 isis kernel: scsi(0): Port database changed.
Dec 16 16:39:19 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:39:19 isis kernel: scsi(0): Port database changed.
Dec 16 16:39:20 isis last message repeated 2 times
Dec 16 16:39:26 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:39:26 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:40:01 isis kernel: qla2x00: Status Entry invalid handle.
Dec 16 16:40:01 isis kernel: qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:40:04 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:40:04 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:40:04 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:40:04 isis kernel: scsi(0): LIP occurred.
Dec 16 16:40:04 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:40:04 isis kernel: scsi(0): Port database changed.
Dec 16 16:40:04 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:40:05 isis kernel: scsi(0): Port database changed.
Dec 16 16:40:05 isis last message repeated 2 times
Dec 16 16:40:11 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:40:11 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:41:42 isis kernel: qla2xxx_eh_abort scsi(0:0:0:11): cmd_timeout_in_sec=0x3c.
Dec 16 16:41:45 isis kernel: qla2x00_ioctl_error_recovery(0) issuing device reset
Dec 16 16:41:45 isis kernel: qla2xxx_eh_device_reset(): **** CMD derives a NULL TGT_Q
Dec 16 16:41:45 isis kernel: qla2x00_ioctl_error_recovery(0) elevation to host_reset
Dec 16 16:41:45 isis kernel: scsi(0:0:255:0): now issue ADAPTER RESET.
Dec 16 16:41:45 isis kernel: qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:41:45 isis kernel: qla2x00_eh_wait_on_command: found cmd=c6852c00.
Dec 16 16:41:47 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:41:47 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:41:47 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:41:47 isis kernel: scsi(0): LIP occurred.
Dec 16 16:41:47 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:41:47 isis kernel: scsi(0): Port database changed.
Dec 16 16:41:59 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:41:59 isis kernel: scsi(0): Port database changed.
Dec 16 16:41:59 isis last message repeated 2 times
Dec 16 16:42:00 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:42:00 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:42:00 isis kernel: qla2xxx_eh_host_reset(0): reset succeded
Dec 16 16:42:00 isis kernel: qla2x00_ioctl_error_recovery(0) return_status=2002
Dec 16 16:42:00 isis kernel: qla2xxx_eh_abort: cmd already done sp=00000000
Dec 16 16:42:00 isis last message repeated 14 times
Dec 16 16:42:00 isis kernel: scsi(0:0:0:11): DEVICE RESET ISSUED.
Dec 16 16:42:00 isis kernel: scsi(0:0:0:11): DEVICE RESET SUCCEEDED.
Dec 16 16:42:32 isis kernel: qla2x00: ISP System Error - mbx1=1c36h, mbx2=2h, mbx3=2h.<6>qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:42:34 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:42:34 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:42:34 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:42:34 isis kernel: scsi(0): LIP occurred.
Dec 16 16:42:34 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:42:34 isis kernel: scsi(0): Port database changed.
Dec 16 16:42:35 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:42:35 isis kernel: scsi(0): Port database changed.
Dec 16 16:42:35 isis last message repeated 2 times
Dec 16 16:42:41 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:42:41 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:45:14 isis kernel: qla2x00_ioctl_error_recovery(0) issuing device reset
Dec 16 16:45:14 isis kernel: qla2xxx_eh_device_reset(): **** CMD derives a NULL TGT_Q
Dec 16 16:45:14 isis kernel: qla2x00_ioctl_error_recovery(0) elevation to host_reset
Dec 16 16:45:14 isis kernel: scsi(0:0:255:0): now issue ADAPTER RESET.
Dec 16 16:45:14 isis kernel: qla2x00(0): Performing ISP error recovery - ha= c6e9c0c0.
Dec 16 16:45:16 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:45:16 isis kernel: scsi(0): Waiting for LIP to complete...
Dec 16 16:45:16 isis kernel: scsi(0): LIP reset occurred.
Dec 16 16:45:16 isis kernel: scsi(0): LIP occurred.
Dec 16 16:45:16 isis kernel: scsi(0): LOOP UP detected.
Dec 16 16:45:16 isis kernel: scsi(0): Port database changed.
Dec 16 16:45:17 isis kernel: scsi(0): Topology - (FL_Port), Host Loop address 0x0
Dec 16 16:45:17 isis kernel: scsi(0): Port database changed.
Dec 16 16:45:24 isis last message repeated 2 times
Dec 16 16:45:24 isis kernel: scsi(0) :Loop id 0x0082 is a MSA1000 device
Dec 16 16:45:24 isis kernel: qla2x00_enable_auto_restore(0) support disabled for MSA/EVA Active/Passive device
Dec 16 16:45:24 isis kernel: qla2xxx_eh_host_reset(0): reset succeded
Dec 16 16:45:24 isis kernel: qla2x00_ioctl_error_recovery(0) return_status=2002
Dec 16 16:45:24 isis kernel: qla2xxx_eh_abort: cmd already done sp=00000000
===================END=========================

while in the other servers start there is a similar message in your logs:

===============================================
Dec 16 16:20:54 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:20:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:20:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:20:54 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:26:57 poseidon -- MARK --
Dec 16 16:29:07 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:29:07 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:29:07 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:29:07 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:36:57 poseidon -- MARK --
Dec 16 16:39:25 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:39:25 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:39:25 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:39:25 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:40:10 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:40:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:40:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:40:10 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:41:54 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:41:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:41:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:41:54 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:42:41 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:42:41 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:42:41 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:42:41 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:45:23 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:45:23 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:45:23 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:45:23 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:49:22 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:49:22 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:49:22 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:49:22 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:49:35 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 16:49:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:49:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 16:49:35 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 16:56:58 poseidon -- MARK --
Dec 16 17:01:35 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:01:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:01:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:01:35 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:06:58 poseidon -- MARK --
Dec 16 17:16:58 poseidon -- MARK --
Dec 16 17:18:47 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:18:47 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:18:47 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:18:47 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:20:31 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5e8.
Dec 16 17:20:31 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:20:31 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:20:31 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:20:31 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:20:35 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5e8.
Dec 16 17:20:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:20:35 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:20:35 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:10 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:21:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:10 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:10 poseidon kernel: scsi(2): RSCN database changed -0x1,0x5ef.
Dec 16 17:21:10 poseidon kernel: scsi(2): Waiting for LIP to complete...
Dec 16 17:21:10 poseidon kernel: scsi(2): Waiting for LIP to complete...
Dec 16 17:21:10 poseidon kernel: scsi(2): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:38 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:21:38 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:38 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:38 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:40 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:21:40 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:40 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:21:40 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:45 poseidon kernel: scsi(3): Port database changed.
Dec 16 17:21:53 poseidon kernel: scsi(2): RSCN database changed -0x1,0x5ef.
Dec 16 17:21:53 poseidon kernel: scsi(2): Waiting for LIP to complete...
Dec 16 17:21:53 poseidon kernel: scsi(2): Waiting for LIP to complete...
Dec 16 17:21:53 poseidon kernel: scsi(2): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:21:59 poseidon kernel: scsi(2): Port database changed.
Dec 16 17:27:40 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:27:40 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:40 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:40 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:27:47 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:27:47 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:47 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:47 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:27:54 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:27:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:54 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:27:54 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:30:25 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:30:25 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:30:25 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:30:25 poseidon kernel: scsi(3): Topology - (F_Port), Host Loop address 0xffff
Dec 16 17:31:10 poseidon kernel: scsi(3): RSCN database changed -0x1,0x5ef.
Dec 16 17:31:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
Dec 16 17:31:10 poseidon kernel: scsi(3): Waiting for LIP to complete...
=====================END=======================

This has complicated me, that is completely random.

please need help

best regards,
Edgard