topic Re: Decoding lbolt errors in Operating System - HP-UX

Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 13:59:47 GMT

Can anyone tell me how to figure out to which device(s) these errors refer?

dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111200, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111201, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111202, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111203, dev: cb056002, io_id: 500076c
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111203, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111204, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111205, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111206, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111207, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111208, dev: cb058002, io_id: 500076d
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111208, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111209, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111210, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111211, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111212, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111213, dev: cb059002, io_id: 500076e
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111213, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111214, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111215, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111216, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111217, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111218, dev: cb05a002, io_id: 500076f
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111218, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111219, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111220, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111221, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111222, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111223, dev: cb05b002, io_id: 5000770
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111223, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111224, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111225, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111226, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111227, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111228, dev: cb05c002, io_id: 5000771
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111228, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111229, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111230, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111231, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111232, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111233, dev: cb05d002, io_id: 5000772
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111233, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111234, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111235, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111236, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111237, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111238, dev: cb05e002, io_id: 5000773
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111238, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111239, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111240, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111241, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111242, dev: cb05f002, io_id: 5000774
dmesg-dbahp-21:SCSI: Unexpected Disconnect -- lbolt: 67111243, dev: cb05f002, io_id: 5000774

Re: Decoding lbolt errors

spex — Thu, 21 Dec 2006 14:12:10 GMT

Hello,

The dev format is:

MMcctdxx (hex)
MM = major dev no.
cc = controller
t = target
d = device

Which turns into
/dev/dsk/c_t_d_

For example:
dev: cb056002
MM = cb (hex) = 203 (dec)
cc = 05
t = 6
d = 0
/dev/dsk/c05t6d0
# lsdev 203
Character Block Driver Class
203 -1 sctl ctl

PCS

Re: Decoding lbolt errors

Michael Steele_2 — Thu, 21 Dec 2006 14:12:20 GMT

cb - 05 - f - 0 - 0 - 2
(cb05f002)

major number = cb or 203
controller = 05
target = 15
lun = 0
slice = 2

c5t15d0s2

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=943878

Try 'pvchange -t 180 /de/dsk/c5t15d0s2'

Re: Decoding lbolt errors

Sandman! — Thu, 21 Dec 2006 14:25:49 GMT

Check out the TKB doc below:

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000080065993

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 14:36:50 GMT

Both suggestions appear to make sense, however I do not have any disks on card 05. Could it be some other type of device?

Re: Decoding lbolt errors

Sandman! — Thu, 21 Dec 2006 14:46:00 GMT

The major device number -> cb (hex) = 203 (dec) which is the SCSI controller driver (sctl) on the system. You may have a bad SCSI HBA connected to either internal or external storage (disks).

Could you post the output of the the command below:

# ioscan -d sctl

Re: Decoding lbolt errors

Michael Steele_2 — Thu, 21 Dec 2006 14:49:37 GMT

Hi Janet:

a) Here's a doc:

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&admit=-682735245+1166730393370+28353475&docId=200000080065993

b) Could you paste in :

# ioscan -fnC ext_bus

c) Can you check 'dmesg' for similar device errors as well as syslog.log?

d) # lsdev | grep 203 -or- # lsdev 203

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 14:54:49 GMT

H/W Path Class Description
===============================================================
0/0/1/0.7.0 ctl Initiator
0/0/1/1.7.0 ctl Initiator
0/0/2/0.7.0 ctl Initiator
0/0/2/1.7.0 ctl Initiator
0/1/3/0.7.0 ctl Initiator
0/2/0/0.106.22.255.1.3.0 ctl DGC CX600
0/2/0/0.106.22.255.1.3.1 ctl DGC CX600
0/2/0/0.106.22.255.1.3.2 ctl DGC CX600
0/2/0/0.106.22.255.1.3.3 ctl DGC CX600
0/2/0/0.106.22.255.1.3.4 ctl DGC CX600
0/2/0/0.106.22.255.1.3.5 ctl DGC CX600
0/2/0/0.106.22.255.1.3.6 ctl DGC CX600
0/2/0/0.106.22.255.1.3.7 ctl DGC CX600
0/2/0/0.106.25.255.1.3.0 ctl DGC CX600
0/2/0/0.106.25.255.1.3.1 ctl DGC CX600
0/2/0/0.106.25.255.1.3.2 ctl DGC CX600
0/2/0/0.106.25.255.1.3.3 ctl DGC CX600
0/2/0/0.106.25.255.1.3.4 ctl DGC CX600
0/2/0/0.106.25.255.1.3.5 ctl DGC CX600
0/2/0/0.106.25.255.1.3.6 ctl DGC CX600
0/2/0/0.106.25.255.1.3.7 ctl DGC CX600
0/4/0/0.7.0 ctl Initiator
0/6/0/0.105.22.255.1.3.0 ctl DGC CX600
0/6/0/0.105.22.255.1.3.1 ctl DGC CX600
0/6/0/0.105.22.255.1.3.2 ctl DGC CX600
0/6/0/0.105.22.255.1.3.3 ctl DGC CX600
0/6/0/0.105.22.255.1.3.4 ctl DGC CX600
0/6/0/0.105.22.255.1.3.5 ctl DGC CX600
0/6/0/0.105.22.255.1.3.6 ctl DGC CX600
0/6/0/0.105.22.255.1.3.7 ctl DGC CX600
0/6/0/0.105.25.255.1.3.0 ctl DGC CX600
0/6/0/0.105.25.255.1.3.1 ctl DGC CX600
0/6/0/0.105.25.255.1.3.2 ctl DGC CX600
0/6/0/0.105.25.255.1.3.3 ctl DGC CX600
0/6/0/0.105.25.255.1.3.4 ctl DGC CX600
0/6/0/0.105.25.255.1.3.5 ctl DGC CX600
0/6/0/0.105.25.255.1.3.6 ctl DGC CX600
0/6/0/0.105.25.255.1.3.7 ctl DGC CX600

I did find some corresponding device files off /dev/dsk but I don't know what they are (or where).

Thanks

Re: Decoding lbolt errors

Sandman! — Thu, 21 Dec 2006 14:58:19 GMT

Correction to my last posting...please post the output of the following:

# ioscan -fun -d sctl
and
# ioscan -funC ext_bus

thanks

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 15:12:42 GMT

pbrun remsh dbahp /usr/sbin/ioscan -fun -d sctl
** Validating Command in New Logic **
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ctl 1 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ctl 2 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ctl 3 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
ctl 4 0/1/3/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
ctl 6 0/2/0/0.106.22.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d0
ctl 7 0/2/0/0.106.22.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d1
ctl 8 0/2/0/0.106.22.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d2
ctl 10 0/2/0/0.106.22.255.1.3.3 sctl CLAIMED DEVICE DGC C
/dev/rscsi/c8t3d3
ctl 12 0/2/0/0.106.22.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d4
ctl 14 0/2/0/0.106.22.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d5
ctl 15 0/2/0/0.106.22.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d6
ctl 17 0/2/0/0.106.22.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c8t3d7
ctl 30 0/2/0/0.106.25.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d0
ctl 31 0/2/0/0.106.25.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d1
ctl 32 0/2/0/0.106.25.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d2
ctl 33 0/2/0/0.106.25.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d3
ctl 34 0/2/0/0.106.25.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d4
ctl 35 0/2/0/0.106.25.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d5
ctl 36 0/2/0/0.106.25.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d6
ctl 37 0/2/0/0.106.25.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c12t3d7
ctl 5 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c5t7d0
ctl 9 0/6/0/0.105.22.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d0
ctl 11 0/6/0/0.105.22.255.1.3.1 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d1

ctl 13 0/6/0/0.105.22.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d2
ctl 16 0/6/0/0.105.22.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d3
ctl 18 0/6/0/0.105.22.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d4
ctl 19 0/6/0/0.105.22.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d5
ctl 20 0/6/0/0.105.22.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d6
ctl 21 0/6/0/0.105.22.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c6t3d7
ctl 22 0/6/0/0.105.25.255.1.3.0 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d0
ctl 23 0/6/0/0.105.25.255.1.3.1 sctl CLAIMED DEVICE DGC CX600

/dev/rscsi/c7t3d1
ctl 24 0/6/0/0.105.25.255.1.3.2 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d2
ctl 25 0/6/0/0.105.25.255.1.3.3 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d3
ctl 26 0/6/0/0.105.25.255.1.3.4 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d4
ctl 27 0/6/0/0.105.25.255.1.3.5 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d5
ctl 28 0/6/0/0.105.25.255.1.3.6 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d6
ctl 29 0/6/0/0.105.25.255.1.3.7 sctl CLAIMED DEVICE DGC CX600
/dev/rscsi/c7t3d7
** Validating Command in New Logic **
pbrun remsh dbahp /usr/sbin/ioscan -funC ext_bus
** Validating Command in New Logic **
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Fast Wide Single-Ended
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra Wide Single-Ended
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C87x Ultra Wide Single-Ended
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C87x Ultra Wide Single-Ended
ext_bus 4 0/1/3/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential
ext_bus 9 0/2/0/0.106.22.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 8 0/2/0/0.106.22.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 13 0/2/0/0.106.25.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 12 0/2/0/0.106.25.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 5 0/4/0/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential
ext_bus 10 0/6/0/0.105.22.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 6 0/6/0/0.105.22.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface
ext_bus 11 0/6/0/0.105.25.19.0 fcparray CLAIMED INTERFACE FCP Array Interface
ext_bus 7 0/6/0/0.105.25.255.1 fcpdev CLAIMED INTERFACE FCP Device Interface

Re: Decoding lbolt errors

Sandman! — Thu, 21 Dec 2006 15:21:33 GMT

Thanks for posting those ioscans. The suspect is:

ext_bus 5 0/4/0/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wide Differential

Posting the output of one more ioscan would help:

# ioscan -I 5 -C disk

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 15:34:13 GMT

Which character is that after the first dash, an l or an I?

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 15:36:18 GMT

Ok, that made no sense, I mean a lower case "L" (as in letter that comes after K)or an upper case "I" (as in letter that come after H)

Re: Decoding lbolt errors

Sandman! — Thu, 21 Dec 2006 15:54:11 GMT

upper case i not a lower case ell

Re: Decoding lbolt errors

spex — Thu, 21 Dec 2006 15:58:15 GMT

Janet,

A quick and easy way to distinguish between an uppercase "I" (char 0x49) and a lowercase "l" (char 0x6c) is to paste the letter in question into vi (or a vi-style shell prompt) and "twiddle" it (tap "~" to change its case).

In the example above, it's an "I" ("eye"), by the way.

PCS

Re: Decoding lbolt errors

Janet Sanberg — Thu, 21 Dec 2006 15:58:30 GMT

Yes, thank you. I wanted to verify I was typing the right command because it returns nothing.

Re: Decoding lbolt errors

Michael Steele_2 — Thu, 21 Dec 2006 16:32:59 GMT

Dear Janet:

Since this isn't showing up in your ioscan I suggest you check the back of the box for properly terminated SCSI connections.

What is the model of the box?

And can you also copy in this script and paste in the report from this script?

/usr/sbin/cstm<<-EOF
runutil logtool
rs
EOF

Re: Decoding lbolt errors

Andrew Merritt_2 — Fri, 22 Dec 2006 09:30:03 GMT

Hi Janet,
One minor thing which the pedant in me won't let pass: these are SCSI errors. The 'lbolt' is a time value, not an error indication, there's no such thing as an 'lbolt error' (well, I suppose if the value shown was incorrect, it might be).

Andrew

Re: Decoding lbolt errors

Janet Sanberg — Fri, 22 Dec 2006 09:32:45 GMT

Thank you all. I have found the offending unterminated SCSI connection and have terminated it. Errors have stopped. Thanks again!!!

Re: Decoding lbolt errors

Janet Sanberg — Fri, 22 Dec 2006 09:33:53 GMT

Terminated SCSi connection