Decoding lbolt errors

[Janet Sanberg]

Can anyone tell me how to figure out to which device(s) these errors refer?

dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111200, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111201, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111202, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111203, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111203, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111204, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111205, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111206, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111207, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111208, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111208, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111209, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111210, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111211, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111212, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111213, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111213, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111214, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111215, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111216, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111217, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111218, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111218, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111219, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111220, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111221, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111222, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111223, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111223, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111224, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111225, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111226, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111227, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111228, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111228, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111229, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111230, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111231, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111232, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111233, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111233, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111234, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111235, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111236, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111237, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111238, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111238, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111239, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111240, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111241, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111242, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111243, dev: cb05f002, io_id: 5000774

[spex]

Hello,

The dev format is:

MMcctdxx (hex)
MM = major dev no.
cc = controller
t = target
d = device

Which turns into
/dev/dsk/c_t_d_

For example:
dev: cb056002
MM = cb (hex) = 203 (dec)
cc = 05
t = 6
d = 0
/dev/dsk/c05t6d0
# lsdev 203
Character Block Driver Class
203 -1 sctl ctl

PCS

[Michael Steele_2]

cb - 05 - f - 0 - 0 - 2
(cb05f002)


major number = cb or 203
controller = 05
target = 15
lun = 0
slice = 2

c5t15d0s2

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=943878

Try 'pvchange -t 180 /de/dsk/c5t15d0s2'

[Sandman!]

Check out the TKB doc below:

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000080065993

[Janet Sanberg]

Both suggestions appear to make sense, however I do not have any disks on card 05. Could it be some other type of device?

[Sandman!]

The major device number -> cb (hex) = 203 (dec) which is the SCSI controller driver (sctl) on the system. You may have a bad SCSI HBA connected to either internal or external storage (disks).

Could you post the output of the the command below:

# ioscan -d sctl

[Michael Steele_2]

Hi Janet:

a) Here's a doc:

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&admit=-682735245+1166730393370+28353475&docId=200000080065993

b) Could you paste in :

# ioscan -fnC ext_bus

c) Can you check 'dmesg' for similar device errors as well as syslog.log?

d) # lsdev | grep 203 -or- # lsdev 203

[Janet Sanberg]

H/W Path Class Description
===============================================================
0/0/1/0.7.0 ctl Initiator
0/0/1/1.7.0 ctl Initiator
0/0/2/0.7.0 ctl Initiator
0/0/2/1.7.0 ctl Initiator
0/1/3/0.7.0 ctl Initiator
0/2/0/0.106.22.255.1.3.0 ctl DGC CX600
0/2/0/0.106.22.255.1.3.1 ctl DGC CX600
0/2/0/0.106.22.255.1.3.2 ctl DGC CX600
0/2/0/0.106.22.255.1.3.3 ctl DGC CX600
0/2/0/0.106.22.255.1.3.4 ctl DGC CX600
0/2/0/0.106.22.255.1.3.5 ctl DGC CX600
0/2/0/0.106.22.255.1.3.6 ctl DGC CX600
0/2/0/0.106.22.255.1.3.7 ctl DGC CX600
0/2/0/0.106.25.255.1.3.0 ctl DGC CX600
0/2/0/0.106.25.255.1.3.1 ctl DGC CX600
0/2/0/0.106.25.255.1.3.2 ctl DGC CX600
0/2/0/0.106.25.255.1.3.3 ctl DGC CX600
0/2/0/0.106.25.255.1.3.4 ctl DGC CX600
0/2/0/0.106.25.255.1.3.5 ctl DGC CX600
0/2/0/0.106.25.255.1.3.6 ctl DGC CX600
0/2/0/0.106.25.255.1.3.7 ctl DGC CX600
0/4/0/0.7.0 ctl Initiator
0/6/0/0.105.22.255.1.3.0 ctl DGC CX600
0/6/0/0.105.22.255.1.3.1 ctl DGC CX600
0/6/0/0.105.22.255.1.3.2 ctl DGC CX600
0/6/0/0.105.22.255.1.3.3 ctl DGC CX600
0/6/0/0.105.22.255.1.3.4 ctl DGC CX600
0/6/0/0.105.22.255.1.3.5 ctl DGC CX600
0/6/0/0.105.22.255.1.3.6 ctl DGC CX600
0/6/0/0.105.22.255.1.3.7 ctl DGC CX600
0/6/0/0.105.25.255.1.3.0 ctl DGC CX600
0/6/0/0.105.25.255.1.3.1 ctl DGC CX600
0/6/0/0.105.25.255.1.3.2 ctl DGC CX600
0/6/0/0.105.25.255.1.3.3 ctl DGC CX600
0/6/0/0.105.25.255.1.3.4 ctl DGC CX600
0/6/0/0.105.25.255.1.3.5 ctl DGC CX600
0/6/0/0.105.25.255.1.3.6 ctl DGC CX600
0/6/0/0.105.25.255.1.3.7 ctl DGC CX600

I did find some corresponding device files off /dev/dsk but I don't know what they are (or where).

Thanks

[Sandman!]

Correction to my last posting...please post the output of the following:

# ioscan -fun -d sctl
and
# ioscan -funC ext_bus

thanks

[Janet Sanberg]

pbrun remsh dbahp /usr/sbin/ioscan -fun -d sctl
** Validating Command in New Logic **
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ctl 1 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ctl 2 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ctl 3 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
ctl 4 0/1/3/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
ctl 6 0/2/0/0.106.22.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d0
ctl 7 0/2/0/0.106.22.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d1
ctl 8 0/2/0/0.106.22.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d2
ctl 10 0/2/0/0.106.22.255.1.3.3 sctl CLAIMED DEVICE DGC C
/dev/rscsi/c8t3d3
ctl 12 0/2/0/0.106.22.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d4
ctl 14 0/2/0/0.106.22.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d5
ctl 15 0/2/0/0.106.22.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d6
ctl 17 0/2/0/0.106.22.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d7
ctl 30 0/2/0/0.106.25.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d0
ctl 31 0/2/0/0.106.25.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d1
ctl 32 0/2/0/0.106.25.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d2
ctl 33 0/2/0/0.106.25.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d3
ctl 34 0/2/0/0.106.25.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d4
ctl 35 0/2/0/0.106.25.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d5
ctl 36 0/2/0/0.106.25.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d6
ctl 37 0/2/0/0.106.25.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d7
ctl 5 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c5t7d0
ctl 9 0/6/0/0.105.22.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d0
ctl 11 0/6/0/0.105.22.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d1

ctl 13 0/6/0/0.105.22.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d2
ctl 16 0/6/0/0.105.22.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d3
ctl 18 0/6/0/0.105.22.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d4
ctl 19 0/6/0/0.105.22.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d5
ctl 20 0/6/0/0.105.22.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d6
ctl 21 0/6/0/0.105.22.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d7
ctl 22 0/6/0/0.105.25.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d0
ctl 23 0/6/0/0.105.25.255.1.3.1 sctl CLAIMED DEVICE DGC CX600

/dev/rscsi/c7t3d1
ctl 24 0/6/0/0.105.25.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d2
ctl 25 0/6/0/0.105.25.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d3
ctl 26 0/6/0/0.105.25.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d4
ctl 27 0/6/0/0.105.25.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d5
ctl 28 0/6/0/0.105.25.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d6
ctl 29 0/6/0/0.105.25.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d7
** Validating Command in New Logic **
pbrun remsh dbahp /usr/sbin/ioscan -funC ext_bus
** Validating Command in New Logic **
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Fast Wide Single-Ended
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide Single-Ended
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C87x Ultra Wide Single-Ended
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C87x Ultra Wide Single-Ended
ext_bus 4 0/1/3/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential
ext_bus 9 0/2/0/0.106.22.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 8 0/2/0/0.106.22.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 13 0/2/0/0.106.25.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 12 0/2/0/0.106.25.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 5 0/4/0/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential
ext_bus 10 0/6/0/0.105.22.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 6 0/6/0/0.105.22.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 11 0/6/0/0.105.25.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 7 0/6/0/0.105.25.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface

[Sandman!]

Thanks for posting those ioscans. The suspect is:

ext_bus 5 0/4/0/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential

Posting the output of one more ioscan would help:

# ioscan -I 5 -C disk

[Janet Sanberg]

Which character is that after the first dash, an l or an I?

[Janet Sanberg]

Ok, that made no sense, I mean a lower case "L" (as in letter that comes after K)or an upper case "I" (as in letter that come after H)

[Sandman!]

upper case i not a lower case ell

[spex]

Janet,

A quick and easy way to distinguish between an uppercase "I" (char 0x49) and a lowercase "l" (char 0x6c) is to paste the letter in question into vi (or a vi-style shell prompt) and "twiddle" it (tap "~" to change its case).

In the example above, it's an "I" ("eye"), by the way.

PCS

[Janet Sanberg]

Yes, thank you. I wanted to verify I was typing the right command because it returns nothing.

[Michael Steele_2]

Dear Janet:

Since this isn't showing up in your ioscan I suggest you check the back of the box for properly terminated SCSI connections.

What is the model of the box?

And can you also copy in this script and paste in the report from this script?

/usr/sbin/cstm<<-EOF
runutil logtool
rs
EOF

[Andrew Merritt_2]

Hi Janet,
One minor thing which the pedant in me won't let pass: these are SCSI errors. The 'lbolt' is a time value, not an error indication, there's no such thing as an 'lbolt error' (well, I suppose if the value shown was incorrect, it might be).

Andrew

[Janet Sanberg]

Thank you all. I have found the offending unterminated SCSI connection and have terminated it. Errors have stopped. Thanks again!!!

[Janet Sanberg]

Terminated SCSi connection