System Panic after Tape library installation

[Karl_]

Hi,

Our server constantly gets system panic after a Compaq MSL5000 Series tape library was installed. It restarted 3 times already today. How can I check if it is really the cause or what caused the system panic.

I suspected the tape library because (1) the system panic did not start until the hardware was installed. (2) the IO scan shows 16 devices for the 2 drive tape library. I think this is not normal.(please see ioscan below). (3) there was no configuration change on the application (SAP) after the hardware was installed.


# ioscan -fnC autoch
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
autoch 0 0/8/0/0.1.10.228.0.0.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t0d2
autoch 1 0/8/0/0.1.10.228.0.1.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t1d2
autoch 2 0/8/0/0.1.10.228.0.2.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t2d2
autoch 3 0/8/0/0.1.10.228.0.3.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t3d2
autoch 4 0/8/0/0.1.10.228.0.4.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t4d2
autoch 5 0/8/0/0.1.10.228.0.5.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t5d2
autoch 6 0/8/0/0.1.10.228.0.6.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t6d2
autoch 7 0/8/0/0.1.10.228.0.7.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t7d2
autoch 8 0/8/0/0.1.10.228.0.8.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t8d2
autoch 9 0/8/0/0.1.10.228.0.9.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t9d2
autoch 10 0/8/0/0.1.10.228.0.10.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t10d2
autoch 11 0/8/0/0.1.10.228.0.11.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t11d2
autoch 12 0/8/0/0.1.10.228.0.12.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t12d2
autoch 13 0/8/0/0.1.10.228.0.13.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t13d2
autoch 14 0/8/0/0.1.10.228.0.14.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t14d2
autoch 15 0/8/0/0.1.10.228.0.15.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c15t15d2
autoch 16 0/8/0/0.1.10.255.14.4.2 schgr CLAIMED DEVICE COMPAQ MSL5000 Series
/dev/rac/c14t4d2


# ioscan -fnC tape
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
tape 1 0/4/0/0.5.0 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/1m /dev/rmt/1mnb /dev/rmt/c1t5d0BESTn
/dev/rmt/1mb /dev/rmt/c1t5d0BEST /dev/rmt/c1t5d0BESTnb
/dev/rmt/1mn /dev/rmt/c1t5d0BESTb
tape 3 0/8/0/0.1.10.228.0.0.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/3m /dev/rmt/3mnb /dev/rmt/c15t0d1BESTn
/dev/rmt/3mb /dev/rmt/c15t0d1BEST /dev/rmt/c15t0d1BESTnb
/dev/rmt/3mn /dev/rmt/c15t0d1BESTb
tape 4 0/8/0/0.1.10.228.0.0.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/4m /dev/rmt/4mnb /dev/rmt/c15t0d3BESTn
/dev/rmt/4mb /dev/rmt/c15t0d3BEST /dev/rmt/c15t0d3BESTnb
/dev/rmt/4mn /dev/rmt/c15t0d3BESTb
tape 5 0/8/0/0.1.10.228.0.1.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/5m /dev/rmt/5mnb /dev/rmt/c15t1d1BESTn
/dev/rmt/5mb /dev/rmt/c15t1d1BEST /dev/rmt/c15t1d1BESTnb
/dev/rmt/5mn /dev/rmt/c15t1d1BESTb
tape 6 0/8/0/0.1.10.228.0.1.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/6m /dev/rmt/6mnb /dev/rmt/c15t1d3BESTn
/dev/rmt/6mb /dev/rmt/c15t1d3BEST /dev/rmt/c15t1d3BESTnb
/dev/rmt/6mn /dev/rmt/c15t1d3BESTb
tape 7 0/8/0/0.1.10.228.0.2.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/7m /dev/rmt/7mnb /dev/rmt/c15t2d1BESTn
/dev/rmt/7mb /dev/rmt/c15t2d1BEST /dev/rmt/c15t2d1BESTnb
/dev/rmt/7mn /dev/rmt/c15t2d1BESTb
tape 8 0/8/0/0.1.10.228.0.2.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/8m /dev/rmt/8mnb /dev/rmt/c15t2d3BESTn
/dev/rmt/8mb /dev/rmt/c15t2d3BEST /dev/rmt/c15t2d3BESTnb
/dev/rmt/8mn /dev/rmt/c15t2d3BESTb
tape 9 0/8/0/0.1.10.228.0.3.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/9m /dev/rmt/9mnb /dev/rmt/c15t3d1BESTn
/dev/rmt/9mb /dev/rmt/c15t3d1BEST /dev/rmt/c15t3d1BESTnb
/dev/rmt/9mn /dev/rmt/c15t3d1BESTb
tape 10 0/8/0/0.1.10.228.0.3.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/10m /dev/rmt/10mnb /dev/rmt/c15t3d3BESTn
/dev/rmt/10mb /dev/rmt/c15t3d3BEST /dev/rmt/c15t3d3BESTnb
/dev/rmt/10mn /dev/rmt/c15t3d3BESTb
tape 11 0/8/0/0.1.10.228.0.4.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/11m /dev/rmt/11mnb /dev/rmt/c15t4d1BESTn
/dev/rmt/11mb /dev/rmt/c15t4d1BEST /dev/rmt/c15t4d1BESTnb
/dev/rmt/11mn /dev/rmt/c15t4d1BESTb
tape 12 0/8/0/0.1.10.228.0.4.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/12m /dev/rmt/12mnb /dev/rmt/c15t4d3BESTn
/dev/rmt/12mb /dev/rmt/c15t4d3BEST /dev/rmt/c15t4d3BESTnb
/dev/rmt/12mn /dev/rmt/c15t4d3BESTb
tape 13 0/8/0/0.1.10.228.0.5.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/13m /dev/rmt/13mnb /dev/rmt/c15t5d1BESTn
/dev/rmt/13mb /dev/rmt/c15t5d1BEST /dev/rmt/c15t5d1BESTnb
/dev/rmt/13mn /dev/rmt/c15t5d1BESTb
tape 14 0/8/0/0.1.10.228.0.5.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/14m /dev/rmt/14mnb /dev/rmt/c15t5d3BESTn
/dev/rmt/14mb /dev/rmt/c15t5d3BEST /dev/rmt/c15t5d3BESTnb
/dev/rmt/14mn /dev/rmt/c15t5d3BESTb
tape 15 0/8/0/0.1.10.228.0.6.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/15m /dev/rmt/15mnb /dev/rmt/c15t6d1BESTn
/dev/rmt/15mb /dev/rmt/c15t6d1BEST /dev/rmt/c15t6d1BESTnb
/dev/rmt/15mn /dev/rmt/c15t6d1BESTb
tape 16 0/8/0/0.1.10.228.0.6.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/16m /dev/rmt/16mnb /dev/rmt/c15t6d3BESTn
/dev/rmt/16mb /dev/rmt/c15t6d3BEST /dev/rmt/c15t6d3BESTnb
/dev/rmt/16mn /dev/rmt/c15t6d3BESTb
tape 17 0/8/0/0.1.10.228.0.7.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/17m /dev/rmt/17mnb /dev/rmt/c15t7d1BESTn
/dev/rmt/17mb /dev/rmt/c15t7d1BEST /dev/rmt/c15t7d1BESTnb
/dev/rmt/17mn /dev/rmt/c15t7d1BESTb
tape 18 0/8/0/0.1.10.228.0.7.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/18m /dev/rmt/18mnb /dev/rmt/c15t7d3BESTn
/dev/rmt/18mb /dev/rmt/c15t7d3BEST /dev/rmt/c15t7d3BESTnb
/dev/rmt/18mn /dev/rmt/c15t7d3BESTb
tape 19 0/8/0/0.1.10.228.0.8.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/19m /dev/rmt/19mnb /dev/rmt/c15t8d1BESTn
/dev/rmt/19mb /dev/rmt/c15t8d1BEST /dev/rmt/c15t8d1BESTnb
/dev/rmt/19mn /dev/rmt/c15t8d1BESTb
tape 20 0/8/0/0.1.10.228.0.8.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/20m /dev/rmt/20mnb /dev/rmt/c15t8d3BESTn
/dev/rmt/20mb /dev/rmt/c15t8d3BEST /dev/rmt/c15t8d3BESTnb
/dev/rmt/20mn /dev/rmt/c15t8d3BESTb
tape 21 0/8/0/0.1.10.228.0.9.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/21m /dev/rmt/21mnb /dev/rmt/c15t9d1BESTn
/dev/rmt/21mb /dev/rmt/c15t9d1BEST /dev/rmt/c15t9d1BESTnb
/dev/rmt/21mn /dev/rmt/c15t9d1BESTb
tape 22 0/8/0/0.1.10.228.0.9.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/22m /dev/rmt/22mnb /dev/rmt/c15t9d3BESTn
/dev/rmt/22mb /dev/rmt/c15t9d3BEST /dev/rmt/c15t9d3BESTnb
/dev/rmt/22mn /dev/rmt/c15t9d3BESTb
tape 23 0/8/0/0.1.10.228.0.10.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/23m /dev/rmt/23mnb /dev/rmt/c15t10d1BESTn
/dev/rmt/23mb /dev/rmt/c15t10d1BEST /dev/rmt/c15t10d1BESTnb
/dev/rmt/23mn /dev/rmt/c15t10d1BESTb
tape 24 0/8/0/0.1.10.228.0.10.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/24m /dev/rmt/24mnb /dev/rmt/c15t10d3BESTn
/dev/rmt/24mb /dev/rmt/c15t10d3BEST /dev/rmt/c15t10d3BESTnb
/dev/rmt/24mn /dev/rmt/c15t10d3BESTb
tape 25 0/8/0/0.1.10.228.0.11.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/25m /dev/rmt/25mnb /dev/rmt/c15t11d1BESTn
/dev/rmt/25mb /dev/rmt/c15t11d1BEST /dev/rmt/c15t11d1BESTnb
/dev/rmt/25mn /dev/rmt/c15t11d1BESTb
tape 26 0/8/0/0.1.10.228.0.11.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/26m /dev/rmt/26mnb /dev/rmt/c15t11d3BESTn
/dev/rmt/26mb /dev/rmt/c15t11d3BEST /dev/rmt/c15t11d3BESTnb
/dev/rmt/26mn /dev/rmt/c15t11d3BESTb
tape 27 0/8/0/0.1.10.228.0.12.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/27m /dev/rmt/27mnb /dev/rmt/c15t12d1BESTn
/dev/rmt/27mb /dev/rmt/c15t12d1BEST /dev/rmt/c15t12d1BESTnb
/dev/rmt/27mn /dev/rmt/c15t12d1BESTb
tape 28 0/8/0/0.1.10.228.0.12.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/28m /dev/rmt/28mnb /dev/rmt/c15t12d3BESTn
/dev/rmt/28mb /dev/rmt/c15t12d3BEST /dev/rmt/c15t12d3BESTnb
/dev/rmt/28mn /dev/rmt/c15t12d3BESTb
tape 29 0/8/0/0.1.10.228.0.13.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/29m /dev/rmt/29mnb /dev/rmt/c15t13d1BESTn
/dev/rmt/29mb /dev/rmt/c15t13d1BEST /dev/rmt/c15t13d1BESTnb
/dev/rmt/29mn /dev/rmt/c15t13d1BESTb
tape 30 0/8/0/0.1.10.228.0.13.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/30m /dev/rmt/30mnb /dev/rmt/c15t13d3BESTn
/dev/rmt/30mb /dev/rmt/c15t13d3BEST /dev/rmt/c15t13d3BESTnb
/dev/rmt/30mn /dev/rmt/c15t13d3BESTb
tape 31 0/8/0/0.1.10.228.0.14.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/31m /dev/rmt/31mnb /dev/rmt/c15t14d1BESTn
/dev/rmt/31mb /dev/rmt/c15t14d1BEST /dev/rmt/c15t14d1BESTnb
/dev/rmt/31mn /dev/rmt/c15t14d1BESTb
tape 32 0/8/0/0.1.10.228.0.14.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/32m /dev/rmt/32mnb /dev/rmt/c15t14d3BESTn
/dev/rmt/32mb /dev/rmt/c15t14d3BEST /dev/rmt/c15t14d3BESTnb
/dev/rmt/32mn /dev/rmt/c15t14d3BESTb
tape 33 0/8/0/0.1.10.228.0.15.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/33m /dev/rmt/33mnb /dev/rmt/c15t15d1BESTn
/dev/rmt/33mb /dev/rmt/c15t15d1BEST /dev/rmt/c15t15d1BESTnb
/dev/rmt/33mn /dev/rmt/c15t15d1BESTb
tape 34 0/8/0/0.1.10.228.0.15.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/34m /dev/rmt/34mnb /dev/rmt/c15t15d3BESTn
/dev/rmt/34mb /dev/rmt/c15t15d3BEST /dev/rmt/c15t15d3BESTnb
/dev/rmt/34mn /dev/rmt/c15t15d3BESTb
tape 35 0/8/0/0.1.10.255.14.4.1 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/35m /dev/rmt/35mnb /dev/rmt/c14t4d1BESTn
/dev/rmt/35mb /dev/rmt/c14t4d1BEST /dev/rmt/c14t4d1BESTnb
/dev/rmt/35mn /dev/rmt/c14t4d1BESTb
tape 36 0/8/0/0.1.10.255.14.4.3 stape CLAIMED DEVICE HP Ultrium 2-SCSI
/dev/rmt/36m /dev/rmt/36mnb /dev/rmt/c14t4d3BESTn
/dev/rmt/36mb /dev/rmt/c14t4d3BEST /dev/rmt/c14t4d3BESTnb
/dev/rmt/36mn /dev/rmt/c14t4d3BESTb


Thanks,

Karl

[luckyxing · Accepted solution]

I think whether the scsi ID is conflict

[A. Clay Stephenson]

It certainly looks like you have way too many robotic devices. I would rmsf all of these and insf again. Make sure that you have no device conflicts before doing the ioscan/insf.

[Jan de Haas_3]

You might just disconnect the library for half a day to see if the server still crashes.

Check if the SCSI bus of the library is properly terminated (and with the proper terminator).

Also check if the library is working properly by moving tapes to and from slots/drives from the front panel.

It looks as if you're using fiber channel to connect to the library. Make sure you use the latest fc patches.

Last but not least, if you've got softwate/hardware support with HP. Drop the crashdump off at HP support to have it analyzed.

Good luck.

[Jan de Haas_3]

After you tested the library by moving tapes around with the frontpanel, you can test if the driver/SCSI bus is working properly by using the 'mc (1m)' command.

mc -p -r DS

Should show all drives and slots.

[Rajeev Shukla]

Hi Karl,

I too had experienced similar problem sometime back. The problem started when i changed the switch port for the HBA on the FC-Switch. After connecting on a different port i found all these nonsence happening..got some 30-40 device files created.
But anyway i didnt few things but not sure which ones fixed the Issue.
1. Switched off the library
2. Did ioscan -fn
3. Deleted all the device file related to library (rmsf)
4. Reset of NSR card to factory defaults and reconfigure (Now this was because of another issue, but not sure if this was also a fix).
5. Turned on the library and did ioscan -fn
6. found only one library and then re-created device files using insf -e

Now just try all steps except for step 4..leave it as a last option.

[Torsten.]

Please check the NSR device mapping.

Should look like this (attached)

[Karl_]

Hi All,

Thanks for your reply. Those information was very useful but unfortunately, our hardware provider said that the problem was in the raid controler. The server is now working okay after the controler was replaced. The number of autoloader devices is still 16. I will bring this up to our hardware provider.

Anyway, 10 points for all.

Thanks

Karl