Failing Disk

[Waqar Razi]

I am seeing the following errors in the syslog: Can some one please guide me in identifying the failing disk. I guess it is c0t6d0 but when I did the vgdisplay and pvdisplay and ioscan, it seems to be fine.

Jun 15 18:17:48 orange vmunix: SCSI: isrEscape Controller at 1/0/0/3/0.
Jun 15 18:17:48 orange vmunix: SCSI: Scripts detected non-active tag -- lbolt:
1609259575, bus: 0
Jun 15 18:17:48 orange vmunix: lbp->state: 30008
Jun 15 18:17:48 orange vmunix: lbp->offset: ffffffff
Jun 15 18:17:48 orange vmunix:
Jun 15 18:17:48 orange vmunix: lbp->nominalOffset: 360
Jun 15 18:17:48 orange vmunix: lbp->Cmdindex: 1d
Jun 15 18:17:48 orange vmunix: lbp->last_nexus_index: 34
Jun 15 18:17:48 orange vmunix: lbp->nexus_index: 35
Jun 15 18:17:48 orange vmunix: uCmdSent: 1d005f40 uNexus_offset
: 200056d4
Jun 15 18:17:48 orange vmunix: last lbp->puStatus [00000000412156b4]:
Jun 15 18:17:48 orange vmunix: 0003000e 00030061 0003004d 00030
056
Jun 15 18:17:48 orange vmunix: next lbp->puStatus [00000000412156c4]:
Jun 15 18:17:48 orange vmunix: 00030033 00030009 0003006a 00030
013
Jun 15 18:17:48 orange vmunix: From most recent interrupt:
Jun 15 18:17:48 orange vmunix: ISTAT: 09, SIST0: 00, SIST1: 00,
DSTAT: 84, DSPS: 00000018
Jun 15 18:17:48 orange vmunix: lsp: 0x0000000000000000
Jun 15 18:17:48 orange vmunix: DSAtbl->host_iocb_index: 1d
Jun 15 18:17:48 orange vmunix: DSAtbl->host_iocb_addr: 20005f40
Jun 15 18:17:48 orange vmunix: scratch_lsp: 0x0000000000000000
Jun 15 18:17:48 orange vmunix: c8xx_iocb [fffffff040400b00]:
Jun 15 18:17:48 orange vmunix: 1d005f40 ff000061 20001840 9f061
f80
Jun 15 18:17:48 orange vmunix: 00000003 20005f20 0000000a 20005
f28
Jun 15 18:17:48 orange vmunix: Pre-DSP script dump [fffffff040400018]:
Jun 15 18:17:48 orange vmunix: 90080000 00000000 e1340004 80400
c28
Jun 15 18:17:48 orange vmunix: e1640004 80400c2c 74360100 00000
000
Jun 15 18:17:48 orange vmunix: Script dump [fffffff040400038]:
Jun 15 18:17:48 orange vmunix: 980c0000 00000018 7a5c0100 00000
000
Jun 15 18:17:48 orange vmunix: 7a360200 00000000 90080000 00000
000
Jun 15 18:17:48 orange vmunix: NCR chip register dump for: 0x400200a
Jun 15 18:17:48 orange vmunix: 00: SCNTL3: 9f SCNTL2: 80 SC
NTL1: 10 SCNTL0: da
Jun 15 18:17:48 orange vmunix: 04: GPREG: 0e SDID: 06 SX
FER: 1f SCID: 47
Jun 15 18:17:48 orange vmunix: 08: SBCL: 67 SSID: 86 SO
CL: 47 SFBR: 00
Jun 15 18:17:48 orange vmunix: 0c: SSTAT2: 09 SSTAT1: 0f SS
TAT0: 01 DSTAT: 80
Jun 15 18:17:48 orange vmunix: 10: DSA: 80400b00
Jun 15 18:17:48 orange vmunix: 14: MBOX1: 00 MBOX0: 00 IS
TAT1: 00 ISTAT: 08
Jun 15 18:17:48 orange vmunix: 1c: TEMP: 80400450
Jun 15 18:17:48 orange vmunix: 24: DCMDDBC: 9c0c0000
Jun 15 18:17:48 orange vmunix: 28: DNAD: 80400038
Jun 15 18:17:48 orange vmunix: 2c: DSP: 80400040
Jun 15 18:17:48 orange vmunix: 30: DSPS: 00000018
Jun 15 18:17:48 orange vmunix: 34: SCRATCHA: 00000028
Jun 15 18:17:48 orange vmunix: 38: DCNTL: a1 DWT: 00 DI
EN: 7f DMODE: 4c
Jun 15 18:17:48 orange vmunix: 3c: ADDER: 80400058
Jun 15 18:17:48 orange vmunix: 40: SIST1: 00 SIST0: 10 SI
EN1: 97 SIEN0: 8f
Jun 15 18:17:48 orange vmunix: 44: GPCNTL: 2f MACNTL: 00 SW
IDE: 00 SLPAR: 00
Jun 15 18:17:48 orange vmunix: 48: RESPID1: 00 RESPID0: 80 ST
IME1: 00 STIME0: fc
Jun 15 18:17:48 orange vmunix: 4c: STEST3: 80 STEST2: 00 ST
EST1: 0c STEST0: 76
Jun 15 18:17:48 orange vmunix: 50: RESV50: 00 RESV51: c0 SI
DL1: 00 SIDL0: 28
Jun 15 18:17:48 orange vmunix: 54: CCNTL1: 01 CCNTL0: 01 SO
DL1: 00 SODL0: 00
Jun 15 18:17:48 orange vmunix: 58: RESV58: 00 RESV59: 00 SB
DL1: 00 SBDL0: 28
Jun 15 18:17:48 orange vmunix: 5c: SCRATCHB: 00060002
Jun 15 18:17:48 orange vmunix: 60: SCRATCHC: c0ffffff
Jun 15 18:17:48 orange vmunix: 64: SCRATCHD: 80400288
Jun 15 18:17:48 orange vmunix: 68: SCRATCHE: 80400c2c
Jun 15 18:17:48 orange vmunix: 6c: SCRATCHF: 20000638
Jun 15 18:17:48 orange vmunix: 70: SCRATCHG: 9f061f1d
Jun 15 18:17:48 orange vmunix: 74: SCRATCHH: 200056d4
Jun 15 18:17:48 orange vmunix: 78: SCRATCHI: 09819f1f
Jun 15 18:17:48 orange vmunix: 7c: SCRATCHJ: 1d005f40
Jun 15 18:17:48 orange vmunix: bc: SCNTL4: 80
Jun 15 18:17:48 orange vmunix: PCI configuration register dump:
Jun 15 18:17:48 orange vmunix: Command: 0157
Jun 15 18:17:48 orange vmunix: Latency Timer: ff
Jun 15 18:17:48 orange vmunix: Cache Line Size: 10
Jun 15 18:18:22 orange vmunix:
Jun 15 18:18:22 orange vmunix: SCSI: Resetting SCSI -- lbolt: 1609259675, bus:
0 path: 1/0/0/3/0
Jun 15 18:18:22 orange vmunix: SCSI: Reset detected -- lbolt: 1609259675, bus:
0 path: 1/0/0/3/0
Jun 15 18:18:22 orange vmunix: SCSI: Unexpected Disconnect -- lbolt: 1609260050
, dev: 1f006000, io_id: bd6567
Jun 15 18:18:22 orange vmunix: SCSI: isrEscape Controller at 1/0/0/3/0.
Jun 15 18:18:22 orange vmunix: SCSI: Scripts detected non-active tag -- lbolt:
1609260050, bus: 0
Jun 15 18:18:22 orange vmunix: lbp->state: 30008
Jun 15 18:18:22 orange vmunix: lbp->offset: ffffffff
Jun 15 18:18:22 orange vmunix:
Jun 15 18:18:22 orange above message repeats 3 times
Jun 15 18:18:22 orange vmunix: lbp->nominalOffset: 360
Jun 15 18:18:22 orange vmunix: lbp->Cmdindex: 13
Jun 15 18:18:22 orange vmunix: lbp->last_nexus_index: 5
Jun 15 18:18:22 orange vmunix: lbp->nexus_index: 6
Jun 15 18:18:22 orange vmunix: uCmdSent: 13005cc0 uNexus_offset
: 20005618
Jun 15 18:18:22 orange vmunix: last lbp->puStatus [0000000041215600]:
Jun 15 18:18:22 orange vmunix: 00030079 00030079 00030079 ff031
600
Jun 15 18:18:22 orange vmunix: next lbp->puStatus [00000000412157f0]:
Jun 15 18:18:22 orange vmunix: 00030053 00030013 00030046 00030
00e
Jun 15 18:18:22 orange vmunix: From most recent interrupt:
Jun 15 18:18:22 orange vmunix: ISTAT: 09, SIST0: 00, SIST1: 00,
DSTAT: 84, DSPS: 00000018
Jun 15 18:18:22 orange vmunix: lsp: 0x0000000000000000
Jun 15 18:18:22 orange vmunix: DSAtbl->host_iocb_index: 13
Jun 15 18:18:22 orange vmunix: DSAtbl->host_iocb_addr: 20005cc0
Jun 15 18:18:22 orange vmunix: scratch_lsp: 0x0000000000000000
Jun 15 18:18:22 orange vmunix: c8xx_iocb [fffffff040400b00]:
Jun 15 18:18:22 orange vmunix: 13005cc0 ff000079 20001000 9f061
f80
Jun 15 18:18:22 orange vmunix: 00000003 20005ca0 0000000a 20005
ca8
Jun 15 18:18:22 orange vmunix: Pre-DSP script dump [fffffff040400018]:
Jun 15 18:18:22 orange vmunix: 90080000 00000000 e1340004 80400
ca8
Jun 15 18:18:22 orange vmunix: e1640004 80400cac 74360100 00000
000
Jun 15 18:18:22 orange vmunix: Script dump [fffffff040400038]:
Jun 15 18:18:22 orange vmunix: 980c0000 00000018 7a5c0100 00000
000
Jun 15 18:18:22 orange vmunix: 7a360200 00000000 90080000 00000
000
Jun 15 18:18:22 orange vmunix: NCR chip register dump for: 0x400200a
Jun 15 18:18:22 orange vmunix: 00: SCNTL3: 9f SCNTL2: 80 SC
NTL1: 10 SCNTL0: da
Jun 15 18:18:22 orange vmunix: 04: GPREG: 0e SDID: 06 SX
FER: 1f SCID: 47
Jun 15 18:18:22 orange vmunix: 08: SBCL: 67 SSID: 86 SO
CL: 47 SFBR: 00
Jun 15 18:18:22 orange vmunix: 0c: SSTAT2: 09 SSTAT1: 07 SS
TAT0: 00 DSTAT: 80
Jun 15 18:18:22 orange vmunix: 10: DSA: 80400b00
Jun 15 18:18:22 orange vmunix: 14: MBOX1: 00 MBOX0: 00 IS
TAT1: 00 ISTAT: 08
Jun 15 18:18:22 orange vmunix: 1c: TEMP: 80400450
Jun 15 18:18:22 orange vmunix: 24: DCMDDBC: 9c0c0000
Jun 15 18:18:22 orange vmunix: 28: DNAD: 80400038
Jun 15 18:18:22 orange vmunix: 2c: DSP: 80400040
Jun 15 18:18:22 orange vmunix: 30: DSPS: 00000018
Jun 15 18:18:22 orange vmunix: 34: SCRATCHA: 0000002c
Jun 15 18:18:22 orange vmunix: 38: DCNTL: a1 DWT: 00 DI
EN: 7f DMODE: 4c
Jun 15 18:18:22 orange vmunix: 3c: ADDER: 80400058
Jun 15 18:18:22 orange vmunix: 40: SIST1: 00 SIST0: 10 SI
EN1: 97 SIEN0: 8f
Jun 15 18:18:22 orange vmunix: 44: GPCNTL: 2f MACNTL: 00 SW
IDE: 00 SLPAR: 00Jun 15 18:18:22 orange vmunix: 48: RESPID1: 00 RESPID0: 80 ST
IME1: 00 STIME0: fc
Jun 15 18:18:22 orange vmunix: 4c: STEST3: 80 STEST2: 00 ST
EST1: 0c STEST0: 76
Jun 15 18:18:22 orange vmunix: 50: RESV50: 00 RESV51: c0 SI
DL1: 00 SIDL0: 2c
Jun 15 18:18:22 orange vmunix: 54: CCNTL1: 01 CCNTL0: 01 SO
DL1: 00 SODL0: 00
Jun 15 18:18:22 orange vmunix: 58: RESV58: 00 RESV59: 00 SB
DL1: 00 SBDL0: 2c
Jun 15 18:18:22 orange vmunix: 5c: SCRATCHB: 00060002
Jun 15 18:18:22 orange vmunix: 60: SCRATCHC: c0ffffff
Jun 15 18:18:22 orange vmunix: 64: SCRATCHD: 80400288
Jun 15 18:18:22 orange vmunix: 68: SCRATCHE: 80400cac
Jun 15 18:18:22 orange vmunix: 6c: SCRATCHF: 20000600
Jun 15 18:18:22 orange vmunix: 70: SCRATCHG: 9f061f13
Jun 15 18:18:22 orange vmunix: 74: SCRATCHH: 20005618
Jun 15 18:18:22 orange vmunix: 78: SCRATCHI: 09819f1f
Jun 15 18:18:22 orange vmunix: 7c: SCRATCHJ: 13005cc0
Jun 15 18:18:22 orange vmunix: bc: SCNTL4: 80
Jun 15 18:18:22 orange vmunix: PCI configuration register dump:
Jun 15 18:18:22 orange vmunix: Command: 0157
Jun 15 18:18:22 orange vmunix: Latency Timer: ff
Jun 15 18:18:22 orange vmunix: Cache Line Size: 10
Jun 15 18:18:22 orange vmunix:
Jun 15 18:18:22 orange vmunix: SCSI: Resetting SCSI -- lbolt: 1609260150, bus:
0 path: 1/0/0/3/0
Jun 15 18:18:22 orange vmunix: SCSI: Reset detected -- lbolt: 1609260150, bus:
0 path: 1/0/0/3/0
Jun 15 18:18:22 orange vmunix: SCSI: Unexpected Disconnect -- lbolt: 1609260475
, dev: 1f006000, io_id: bd656f
Jun 15 18:18:22 orange vmunix: SCSI: Ultra160 Controller at 1/0/0/3/0: Error: T
he domain validation test for target 6 determined that communication may not be
possible to this target. Verify the hardware configuration.
Jun 15 18:18:22 orange vmunix: SCSI: isrEscape Controller at 1/0/0/3/0.
Jun 15 18:18:22 orange vmunix: SCSI: Scripts detected non-active tag -- lbolt:
1609260475, bus: 0
Jun 15 18:18:22 orange vmunix: lbp->state: 30008
Jun 15 18:18:22 orange vmunix: lbp->offset: ffffffff
Jun 15 18:18:22 orange vmunix:
Jun 15 18:18:23 orange above message repeats 3 times
Jun 15 18:18:22 orange vmunix: lbp->nominalOffset: 360
Jun 15 18:18:22 orange vmunix: lbp->Cmdindex: f
Jun 15 18:18:22 orange vmunix: lbp->last_nexus_index: 5
Jun 15 18:18:22 orange vmunix: lbp->nexus_index: 0
Jun 15 18:18:22 orange vmunix: uCmdSent: 1005840 uNexus_offset:
0
Jun 15 18:18:22 orange vmunix: last lbp->puStatus [0000000041215600]:
Jun 15 18:18:22 orange vmunix: 00000000 00000000 00000000 00000
000
Jun 15 18:18:22 orange vmunix: next lbp->puStatus [00000000412157f0]:
Jun 15 18:18:22 orange vmunix: 00030053 00030013 00030046 00030
00e
Jun 15 18:18:22 orange vmunix: From most recent interrupt:
Jun 15 18:18:22 orange vmunix: ISTAT: 29, SIST0: 00, SIST1: 00,
DSTAT: 84, DSPS: 00000018
Jun 15 18:18:22 orange vmunix: lsp: 0x0000000000000000
Jun 15 18:18:22 orange vmunix: DSAtbl->host_iocb_index: f
Jun 15 18:18:22 orange vmunix: DSAtbl->host_iocb_addr: 20005840
Jun 15 18:18:22 orange vmunix: scratch_lsp: 0x0000000000000000
Jun 15 18:18:22 orange vmunix: c8xx_iocb [fffffff040400b00]:
Jun 15 18:18:22 orange vmunix: 01005840 ff006079 20001040 77060
000
Jun 15 18:18:22 orange vmunix: 00000007 20005820 00000006 20005
828
Jun 15 18:18:22 orange vmunix: Pre-DSP script dump [fffffff040400018]:
Jun 15 18:18:22 orange vmunix: 90080000 00000000 e1340004 80400
d78
Jun 15 18:18:22 orange vmunix: e1640004 80400d7c 74360100 00000
000
Jun 15 18:18:22 orange vmunix: Script dump [fffffff040400038]:

[Durvesh Mendhekar · Accepted solution]

Hi,

have you checked the /var/opt/resmon/log/event.log

Regards,
Durvesh

[Sunny123_1]

Hi

Check lvdisplay output and look out that any stale extents are there or not???

Regards
Sunny

[Vishu]

Hi Waqar,

Seems, the lbolt error in your syslog caused due to scsi reset. so, you can use the following commands to check the disk in Question :-

1. Ioscan -fnC disk
2. pvdisplay /dev/dsk/
3. diskinfo /dev/rdsk/
4. dd if=/dev/dsk/ of=/dev/null bs=1024k
5. echo 2400?20X | adb /dev/dsk/

if all the outputs ok, then your disk in good.

[Steven E. Protter]

Shalom,

lbolt means one of two things.

* a hot swap disk was replaced and the message will clear next time you boot the system. You might be able to clear it with dmesg -c or dmesg - one of those works on Linux the other on HP-UX and I mix them up.

* A disk is failing. Back up the data if you can and go ahead and replace it now.

SEP

[Vishu]

yes, Steven is right.

lbolt tells the failure of the disk. and you can identify that failure with the above said commands.

[sujit kumar singh]

hi


the messages tell you dev: 1f006000
#ll /dev/dsk | grep "1f006000"
#ll /dev/rdsk | grep "1f006000"

find out the /dev/dsk/cxtydz

do a pvdisplay -v /dev/dsk/cxtydz

note the VG this belongs to and also note the LVs that this disk caters.

if those LV s are mirrored with Strict mirror policy then data is somewhat safe, else take a data backup if possible.

this indicates more probably disk is failing or as said hot swap of a disk could have caused this
regards
sujit