dmesg and syslog warning scsi_changer

[Godfrey Zhang]

Recently I've noticed some scsi_changer warnings coming from dmesg and syslog.log, not don't know what devices they meant. Does Anybody have a clue and how to explain it?

gzhang@tsmgd1:/var/adm/syslog>dmesg

Apr 5 10:31
...
scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0 0 0 0

NOTICE: scsi_changer@28,12,0: scsi_cmd=16,errno=5,cc=300,sense=70 0 6 0 0 0 0 c 0 0 0 0 28 1 0 0 0 0 40 0

gvid: ATI Radeon 7000 not supported on this hardware
gzhang@tsmgd1:/var/adm/syslog>tail -n 200 syslog.log | more
Apr 4 22:42:21 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 4 22:46:21 tsmgd1 above message repeats 187 times
Apr 4 23:00:15 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 4 23:00:15 tsmgd1 vmunix:
Apr 4 23:00:15 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 4 23:04:12 tsmgd1 vmunix:
Apr 4 23:06:21 tsmgd1 above message repeats 335 times
Apr 4 23:04:12 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 4 23:06:21 tsmgd1 above message repeats 47 times
Apr 4 23:04:12 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 4 23:06:21 tsmgd1 above message repeats 287 times
Apr 5 00:00:16 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 00:00:16 tsmgd1 vmunix:
Apr 5 00:00:16 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 00:04:13 tsmgd1 vmunix:
Apr 5 00:06:22 tsmgd1 above message repeats 335 times
Apr 5 00:04:13 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 00:06:22 tsmgd1 above message repeats 47 times
Apr 5 00:04:13 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 00:06:22 tsmgd1 above message repeats 287 times
Apr 5 01:00:15 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 01:00:15 tsmgd1 vmunix:
Apr 5 01:00:15 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=17,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 01:04:11 tsmgd1 vmunix: WARNING: scsi_changer@28,12,0: scsi_cmd=16,errno=201,cc=203,sense=70 0 2 0 0 0 0 c 0 0 0 0 4 3 0 0 0
0 0 0
Apr 5 01:06:22 tsmgd1 above message repeats 287 times
Apr 5 01:04:11 tsmgd1 vmunix:
Apr 5 01:06:22 tsmgd1 above message repeats 335 times

[Rajeev Shukla]

Hi Godfrey,

Either of the SCSI device is doing a SCSI reset which is triggering these logs.
I would suggest you to load "Event Monitoring Service" or the complete online diagnostics. This logs any hardware failure.
This will tell you which SCSI device is actually causing problem.

Cheers
Rajeev

[Stephen Keane]

Can you post your ioscan output?

[Godfrey Zhang]

EMS msg showed one of the FC lost connection causing all SAN disks gone. However, when I check the FC, all back to normal. But this happens nearly every day. No one could figure out what is the culprit.

The box is connected EMC disk array, and installed powerpath:

I would have thought that's /dev/td1, but couldn't figure how this happened, thus couldn't come out a solution to fix it.

nscjpdev:/home/gzhang >powermt display
Symmetrix logical device count=91
CLARiiON logical device count=88
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
22 1/12/0/0.4.23.0.0 optimal 89 0 - 0 0
24 1/12/0/0.4.23.0.1 optimal 2 0 - 0 0
36 1/12/0/0.6.0.0.0 optimal 11 0 - 0 0
37 1/12/0/0.6.16.0.0 optimal 11 0 - 0 0
41 1/12/0/0.2.29.0.0 optimal 77 0 - 0 0
42 1/12/0/0.2.30.0.0 optimal 77 0 - 0 0

nscjpdev:/home/gzhang >ioscan -funC fc
Class I H/W Path Driver S/W State H/W Type Description
=================================================================
fc 0 1/10/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td0
fc 1 1/12/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
/dev/td1
nscjpdev:/home/gzhang >

EMS log, syslog, ioscan log attached.

[Devender Khatana · Accepted solution]

Hi,

It is an abnormal situation and solution might include a few steps in sequence to find out the correct part. As indicated in event log and EMS notifications this might be a link problem. It can be a cable issue. Just see the fc card statics though fcmsutil.i.e. fcmsutil /dev/td0 etc. Furthermore when you have two paths try to remove the path which is displaying continuous errors through fcmsutil, and then let is be running for a day or so. ( Atlease for the time in which frequency you experience this problem). It might be a cable issue. Just simply try by replacing it first if you have some spare with you.

HTH,
Devender