Error in syslog - vmunix: SCSI: Resetting SCSI -- lbolt: 292260891, bus: 0

[Andy Miller_4]

Hi there,

We are having the following errors being reported on our HP 9000 server L2000 server in /var/adm/syslog/syslog.log............


Oct 27 08:14:54 sul7it11 vmunix: SCSI: Resetting SCSI -- lbolt: 248946229, bus: 0

Oct 27 08:14:54 sul7it11 vmunix: SCSI: Reset detected -- lbolt: 248946229, bus: 0

Oct 29 02:02:37 sul7it11 vmunix: scb->cdb: 28 00 00 1d 62 e0 00 00 10 00

Oct 29 02:02:38 sul7it11 vmunix: SCSI: Resetting SCSI -- lbolt: 263993191, bus: 0

Oct 29 02:02:38 sul7it11 vmunix: SCSI: Reset detected -- lbolt: 263993191, bus: 0

Oct 29 05:01:40 sul7it11 vmunix: scb->cdb: 2a 00 00 00 93 30 00 00 10 00

Oct 29 05:01:40 sul7it11 vmunix: scb->cdb: 28 00 00 28 ba d0 00 00 10 00

Oct 29 05:01:41 sul7it11 vmunix: SCSI: Resetting SCSI -- lbolt: 265067591, bus: 0

Oct 29 05:01:41 sul7it11 vmunix: SCSI: Reset detected -- lbolt: 265067591, bus: 0

Nov 1 07:33:42 sul7it11 vmunix: scb->cdb: 28 00 00 ae 03 a0 00 00 50 00

Nov 1 07:33:43 sul7it11 vmunix: SCSI: Resetting SCSI -- lbolt: 292260891, bus: 0

Nov 1 07:33:43 sul7it11 vmunix: SCSI: Reset detected -- lbolt: 292260891, bus: 0

I have seen a similar sort or message before where a disk has has a hardware problem, but the message has always had the hard ware device name also (eg c1t0d0) in the message.


Can anyone out there give me any pointers as to what may possibly be wrong/other things that i can check before something goes majorily wrong please ?

Many thanks,

Sean Harrodine

[Steven E. Protter]

Posbble causes:

You recently did a swap out of a hot swap disk.

A disk has failed
A disk is failing
A disk will fail

Back up the data if any and prepare to identify and replace the disk.

These errors can also be caused by bad cabling and drive cages (certain server models) trashing the disks.

If your box starts eating disks regularly, check those components.

Good test tools:
cstm
mstm
xstm (X win)

SEP

[Geoff Wild]

Do a search on Google - not too sure why ITRC search turns up nothing...

Here's some posts I found:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=523283

Most seem to point to hardware failure...

Rgds...Geoff

[Sridhar Bhaskarla]

Hi Sean,

I believe "bus 0" is what appears to be the one associated with the 'instance 0' of sctl driver that appears in your 'ioscan -fnk' output. Run

/usr/sbin/ioscan -fnk |grep sctl

The path that appears with instance '0' is getting reset. Check what is connected to that bus. Make sure that SCSI bus is properly terminated.

-Sri

[Sanjay_6]

Hi,

Did someone power down / resetted power on a device that is on bus 0 (c0). IT is possible a device was powered off / on or a device disconneted from the bus when the system was still on and the bus was active.

Hope this helps.

Regds

[Andy Miller_4]

Thanks for your responses guys..........

A bit more information :

# /usr/sbin/ioscan -fnk |grep sctl
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
ctl 1 0/0/1/0.14.0 sctl CLAIMED DEVICE HP A5272A
ctl 2 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
ctl 3 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
ctl 4 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
ctl 5 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
ctl 6 0/4/0/0.14.0 sctl CLAIMED DEVICE HP A5272A

[Andy Miller_4]

And a bit more info......

# ioscan -fn
Class I H/W Path Driver S/W State H/W Type Description
============================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (582)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 0 0/0/0/0 btlan3 CLAIMED INTERFACE HP PCI 10/100Base-TX Core
/dev/ether0
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide LVD
target 0 0/0/1/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
target 1 0/0/1/0.8 tgt CLAIMED DEVICE
disk 0 0/0/1/0.8.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c0t8d0 /dev/rdsk/c0t8d0
target 2 0/0/1/0.9 tgt CLAIMED DEVICE
disk 1 0/0/1/0.9.0 sdisk CLAIMED DEVICE IBM DMVS18D
/dev/dsk/c0t9d0 /dev/rdsk/c0t9d0
target 3 0/0/1/0.10 tgt CLAIMED DEVICE
disk 2 0/0/1/0.10.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c0t10d0 /dev/rdsk/c0t10d0
target 4 0/0/1/0.11 tgt CLAIMED DEVICE
disk 3 0/0/1/0.11.0 sdisk CLAIMED DEVICE SEAGATE ST318203LC
/dev/dsk/c0t11d0 /dev/rdsk/c0t11d0
target 5 0/0/1/0.14 tgt CLAIMED DEVICE
ctl 1 0/0/1/0.14.0 sctl CLAIMED DEVICE HP A5272A
/dev/rscsi/c0t14d0
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide Single-Ended
target 6 0/0/1/1.2 tgt CLAIMED DEVICE
disk 4 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
target 7 0/0/1/1.7 tgt CLAIMED DEVICE
ctl 2 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 8 0/0/2/0.7 tgt CLAIMED DEVICE
ctl 3 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C875 Ultra Wide Single-Ended
target 9 0/0/2/1.2 tgt CLAIMED DEVICE
disk 5 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
target 10 0/0/2/1.7 tgt CLAIMED DEVICE
ctl 4 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
tty 0 0/0/4/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag1 /dev/mux0 /dev/tty0p1
/dev/diag/mux0 /dev/tty0p0 /dev/tty0p2
tty 1 0/0/5/0 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag2 /dev/diag/mux1 /dev/mux1 /dev/tty1p1
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 3 0/3 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 4 0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ext_bus 4 0/4/0/0 c720 CLAIMED INTERFACE SCSI C895 Ultra2 Wide LVD
target 11 0/4/0/0.7 tgt CLAIMED DEVICE
ctl 5 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
target 12 0/4/0/0.8 tgt CLAIMED DEVICE
disk 6 0/4/0/0.8.0 sdisk CLAIMED DEVICE SEAGATE ST318203LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
target 13 0/4/0/0.9 tgt CLAIMED DEVICE
disk 7 0/4/0/0.9.0 sdisk CLAIMED DEVICE SEAGATE ST118202LC
/dev/dsk/c4t9d0 /dev/rdsk/c4t9d0
target 14 0/4/0/0.10 tgt CLAIMED DEVICE
disk 8 0/4/0/0.10.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c4t10d0 /dev/rdsk/c4t10d0
target 15 0/4/0/0.11 tgt CLAIMED DEVICE
disk 9 0/4/0/0.11.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c4t11d0 /dev/rdsk/c4t11d0
target 16 0/4/0/0.12 tgt CLAIMED DEVICE
disk 10 0/4/0/0.12.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c4t12d0 /dev/rdsk/c4t12d0
target 17 0/4/0/0.14 tgt CLAIMED DEVICE
ctl 6 0/4/0/0.14.0 sctl CLAIMED DEVICE HP A5272A
/dev/rscsi/c4t14d0
ba 5 0/5 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 6 0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 7 0/7 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 1 0/7/0/0 btlan5 CLAIMED INTERFACE HP A5230A/B5509BA PCI 10/100Base-TX Addon
memory 0 8 memory CLAIMED MEMORY Memory
processor 0 160 processor CLAIMED PROCESSOR Processor
processor 1 162 processor CLAIMED PROCESSOR Processor
processor 2 166 processor CLAIMED PROCESSOR Processor
#


How else/what other commands can i issue to nail the offending device ?

Thanks for all the assistance so far guys.....

rgds,
Sean

[Sanjay_6]

Hi Andy,

IS the bus terminated properly. Tell us a little more about your setup. Is this part of a SG cluster / shared bus or something ?.

Do you see this error message when the system boot / all the time. anything about the frequency of these errors and what happens when you get these errors?.

Thanks

[Andy Miller_4]

Hi Sanjay,

Nothing special about this server.
Its a HP9000 L2000 server with a disk array attached.
The error messages detailed at the top of this fault are the only messages that i have seen in the syslog so as far as i know, this has not been a regular occurence, but as you can see, the first occurence was the 27th oct, then the 29th, and then the 1st Nov.........seems to be getting more regular now !

Regarding terminators, there have been no changes to the server so it should be terminated as normal.

thanks,
Sean

[Sridhar Bhaskarla]

Hi Sean,

The problem seems to be on the following

ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide LVD

If there is any issue with the devices associated with it, you should see POWERFAILED etc., messages. Try running "stm" and see if it reports any errors. Also check the termination.

-Sri

[Andy Miller_4]

Unfortunately, we dont appear to have stm installed. Is ther anything else i can try ?

Sean

[Sanjay_6]

Hi Sean,

I don't think there is much we can help with. How are you on the patches. Can you try and get the latest scsi patches for the version of OS you have on the system. Atleast all the SCSI patches. If that doen're resolve your problem, i would suggest a card replacement.

Hope this helps.

regds

[Victor BERRIDGE]

Hi Andy,
Do you have internal system disks mirrored? and is there some devices sharing one of the controllers used by the mirrored disk?
I had the case where one of the system disk was faulty (but not dead..) playing hide-and seek... with a disk subsystem connected to the same controller, from the controller I could see huge amount of scsi resets...Each time I tried to diagnose, even with EMS the disks were working fine...
took out one mirror disk and the system crashed the next day - and the day after...
and again till I put in the other disk and removed the one in. It ended my trouble and I just had to order a new hotswap disk...


All the best
Victor

[Dave Unverhau_1]

Sean,

Lacking the online diagnostics, you could try dumping each of the disks on the 0/0/1/0 path to the "bit bucket" using the dd command, checking for new timeout entries in the syslog after each drive is dumped. This isn't a conclusive test, but it has proven to be useful in the past.

Use the command similar to:
dd if=/dev/rdsk/cxtydz of=/dev/null bs=8192k

if the command completes with no errors and no new timeout entries in the log, chances are that the drive is healthy.

If you adjust the blocksize (bs) parm to speed up the dump, be sure to specify a multiple of the sector size of the disk.

I hope this helps to isolate the bad boy...

Best Regards,

Dave

[Andy Miller_4]

Hi guys,

Thanks for all your help yesterday.

I have now installed STM so can anyone point me in the right direction as to what i need to run and what i should be looking for ?

TIA

Sean

[Andy Miller_4]

Hi there,

I have issued and save the results of the Information Log from STM for the whole system and would appreciate someone more experienced having a quick look to see what they think the problem could be......

is it the IBM disk on the way out as this appears to be the only device showing any errors ???

TIA
Sean

[Dave Unverhau_1]

Sean, that would be my bet.

I would use STM to exercise the drive (you can do this in read-only mode and not affect the data in any way). After the exercise completes, rerun the info utility on that drive and see if the errors have increased. If so, it's probably time to replace that puppy.

Before replacing it, though, I would back up all the data on the drive (is it mirrored?). Then you might want to run a read/write test on the drive. If the errors don't increase with a read/write test, you might simply have a glitch in the data on the drive that keeps getting read. However, since your errors appear to be only timeouts and you aren't seeing data errors reported per se, I would expect the problem is more likely a glitch in the servo code or an electro-mechanical problem with the drive.

Best Regards,

Dave

[Andy Miller_4]

Cheers Dave,

This disk is mirrored as its our database disk which does periodically get thrashed by poor user queries so i suppose it could be dodgy-ish.

I will give your ideas a twirl and let yuou know....

TIA

Sean