Disk hardware path: I/O request failed

[Yogeeraj_1]

Dear experts,

I have recently connected a disk array that was previously used on my SUN Server E250 (now not being used) to my rp5400(l1000). The interface is SCSI.

This is how the server found it:
# ioscan -fknC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 5 0/0/1/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39102LCSUN9.0G
/dev/dsk/c0t1d0 /dev/rdsk/c0t1d0
disk 6 0/0/1/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39102LCSUN9.0G
/dev/dsk/c0t2d0 /dev/rdsk/c0t2d0
disk 7 0/0/1/0.3.0 sdisk CLAIMED DEVICE SEAGATE ST39102LCSUN9.0G
/dev/dsk/c0t3d0 /dev/rdsk/c0t3d0
disk 8 0/0/1/0.4.0 sdisk CLAIMED DEVICE SEAGATE ST39102LCSUN9.0G
/dev/dsk/c0t4d0 /dev/rdsk/c0t4d0
disk 2 0/0/1/1.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 3 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 4 0/0/2/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 0 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 1 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
#
===============================================

Looked OK but now my EMS is reporting lots of errors, type:
/storage/events/disks/default/0_0_1_0.1.0 is >= 3.
Its current value is CRITICAL(5).
Event data from monitor:

Event Time..........: Tue Feb 1 18:23:47 2005
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 13
System..............: l1000.myserver.mu

Summary:
Disk at hardware path 0/0/1/0.1.0 : I/O request failed.

=========================

Also, on my last reboot, the following errors poped up:
==========================
SCSI: Resetting SCSI -- lbolt: 6753440, bus: 0
SCSI: Reset detected -- lbolt: 6753440, bus: 0
SCSI: First party detected bus hang -- lbolt: 6755243, bus: 0
lbp->state: 5060
lbp->offset: f0
lbp->uPhysScript: f87bc000
From most recent interrupt:
ISTAT: 01, SIST0: 00, SIST1: 00, DSTAT: 84, DSPS: 00000004
lsp: 0000000000000000
lbp->owner: 0000000042c0be00
bp->b_dev: cb001000
scb->io_id: 8d7
scb->cdb: 00 00 00 00 00 00
lbolt_at_timeout: 6754043, lbolt_at_start: 6754043
lsp->state: 10d
scratch_lsp: 0000000000000000
Pre-DSP script dump [fffffffff87bc0f0]:
48000000 00000000 78350000 00000000
50000000 f87bc028 80000000 0000000b
Script dump [fffffffff87bc110]:
0f000001 f87bc5c0 60000040 00000000
9f030000 0000000e 0f000001 f87bc5c1
SCSI: Resetting SCSI -- lbolt: 6755343, bus: 0
SCSI: Reset detected -- lbolt: 6755343, bus: 0
===============================================

I think it is a problem with the firmware of the disks in the enclosure.

what are you views and how do i troubleshoot this?

thank you in advance for your replies.

best regards
Yogeeraj

[T G Manikandan]

Most of the lbolt scsi errors are because of scsi cabling and termination.


Make sure your scsi cabling are right and it is terminated.

[Yogeeraj_1]

Thank TG for your reply.

the cables have been tested OK.

is there a need to upgrade the firmware?

regards
yogeeraj

[Yogeeraj_1]

attached the EMS notification message.

[Shantaram Sahyadri]

Hi,

As stated in a earliear post, ibolt errors relate to scsi cables or termination problems. before identfying firmware updates can u throw some info what what storate box it is, a multipack, or A1000, A5000 etc...

Cheers
Shantaram

[T G Manikandan]

Make sure that you are using the right scsi terminator.

[Yogeeraj_1]

hi,

thank you both for your replies.

1. The storage box is a SUN ultra scsi

2. the terminator on the storage box is a good one and has never been removed from the storage box.

regards
yogeeraj

[T G Manikandan]

What is the model of the disk array that you are using?

[Shantaram Sahyadri]

Hi,

As far as my personal experince goes, ibolt errors do no occur only because of terminators but also beacuse loose held cables, or bad connectors.

well with the info given i belive there is no need to update firmware. Recheck the connecton cables.

Cheers
Shantaram

[Mohanasundaram_1]

Hi Yogeeraj,

The message

SCSI: First party detected bus hang -- lbolt: 6755243, bus: 0

is an indication that the problem is from the controller end. The most plausible cause could be a wrong type of termination.

Have you checked the HBA is compatible with the SCSI technology supported by your storage?

What about the SCSI cable length? If possible can you provide the HBA Part number and the SCSI cable + terminator part number here.

With regards,
Mohan.

[Eric Antunes]

Hi Yogeeraj,

About your firmware question, I know that it will depend of each disk version. I have here at site:

SEAGATEST136403LC - firmware HP04

SEAGATEST336704LC - firmware HP03

SEAGATEST39103LC - firmware HP03

So, I'll need do check with HP, what are the firmaware needs for each of your disks...

Best Regards,

Eric Antunes

[Eric Antunes]

Correction:

SEAGATEST39103LC - firmware HP04

[Eric Antunes]

Yogeeraj,

Check also that you have at least the same firmware version at the disk enclosure that the greatest disk firmware version (in my case I've HP05 on the enclosure witch is greater than the greatest disk firmware version - HP04)

Eric Antunes

[Yogeeraj_1]

hi,

thanks again for your replies.

About the storage box, i cannot decipher the exact model.

Visible on the storage box are the following:

on the front: "Ultra Scsi" + Sun Microsystems
back: PN: 559-2262-01

The SCSI connections looks ok! what should i do?

thanks
regards
yogeeraj

[Mohanasundaram_1]

Hi Yogeeraj,

I have not worked much on SUN Storages and I can't find any reference at SUN's website for that part number you gave.

But before I go for firmware updates, I would always try to make sure that I have got the right combinations of the hardware.

If you do not do this, firmware update is not going to help. Remember, some SCSI cables are physically similar but carry different signals. So, just because you can connect the cables to the storage and SCSI HBA does not mean it is the right kind of cable.

Sorry if I couldn't help much.

With regards,
Mohan.

[Eric Antunes]

Hi,

Check if you have any bad hardware:

#ioscan -fn|grep NO_HW

Did you checked the information of disks, enclosure and everything else on stm? Is it everything green??

Best Regards,

Eric Antunes

[T G Manikandan]

the messages give error for the disk

disk 5 0/0/1/0.1.0 sdisk CLAIMED DEVICE SEAGATE ST39102LCSUN9.0G
/dev/dsk/c0t1d0 /dev/rdsk/c0t1d0

Please remove this disk from the array and revert back on extra messages.

[Yogeeraj_1]

hi again,

ioscan -fn|grep NO_HW
isn't giving me anything!

Just for your information, i have been able to test the disks by creating a volume group and mounting a file system on it!

It was when i first connected the storage box to the server.

the day after, i even installed Oracle 10g and configured it's disk groups to used the storage area (ASM)

and then the day after i noticed those errors!

any clues?

regards
yogeeraj

[Eric Antunes]

Yogeeraj,

Did you see the statuses on stm (XTerm) or mstm?

Eric Antunes

[Yogeeraj_1]

hi Eric,

did "Information" in stm.

posting the before image of stm

regards
yogeeraj

[Yogeeraj_1]

now the after image.


do you think this will fix the problem?

[Eric Antunes]

I don't know if it will correct the EMS messages but it's a possibility.

Did you see your firmware version? Are they HP... (since those disks was from a sun server...)

[Yogeeraj_1]

hi again,

the firmware can't be HP, it is a SUN device!

below the report (STM->Information) about one of the disk:
=============================================
-- Information Tool Log for SCSI Disk on path 0/0/1/0.2.0 --

Log creation time: Wed Feb 2 16:24:53 2005

Hardware path: 0/0/1/0.2.0


The selected device is not supported by this tool.
Inaccurate information could be displayed by the tool.

Product Id: ST39102LCSUN9.0G Vendor: SEAGATE
Device Type: SCSI Disk Firmware Rev: 6329
Device Qualifier: SEAGATEST39102LCSUN9.0G Logical Unit: 0
Serial Number: LJT6161300001950JLW0
Capacity (M Byte): 8637.34
Block Size: 512
Max Block Address: 17689266
Error Logs
Total Retries: 0 Buffer Overruns: N/A
Read Reverse Errors: N/A Buffer Underruns: N/A
Write Errors: 0 Non-Medium Errors: 0
Verify Errors: 0
=============================================

Note that all the other internal disks are also same brand (SEAGATE)

i guess i will have to bounce the server again and wait for the STM to complain again (if ever).

luckily, this is a test server.

Any more insights?

regards
Yogeeraj

[Gerhard Roets]

Hi Yogeeraj

1. Reseat all the cables.
2. Reseat all the terminators.
3. Reseat all the discs in the storage device.

This is where I would start looking before getting a bit to wild about other options.

[Yogeeraj_1]

hi,

i will be doing the reseating tomorrow morning.

Below an extract from my: /etc/opt/resmon/log/api.log

===============================================
-------------------Start Event--------------------
User event occurred at Fri Feb 4 04:53:21.884414 2005
Process ID: 2655 (/usr/sbin/stm/uut/bin/tools/.../disk_em) Log Level: Error
The device answered selection but the command did not complete.

Possible Cause(s)/Recommended Action(s)

The request is not supported by the device.

Incorrect specification of the SCSI command descriptor block, particularly th
e command length. This is an internal application error.

The device took too long or a communication failure occurred. Rerun the tool.
-------------------End Event----------------------

-------------------Start Event--------------------
User event occurred at Fri Feb 4 05:57:12.294493 2005
Process ID: 2655 (/usr/sbin/stm/uut/bin/tools/.../disk_em) Log Level: Error
An attempt to open the device file (/var/tmp/rdskXMBa02655) failed. errno (5) w
as returned by the open system call.

The errno returned from the open system call was unexpected.

Possible Cause(s)/Recommended Action(s):

The device may be offline or not available. Run "ioscan -f" to check the stat
e of the device.

Internal application error.
-------------------End Event----------------------
=============================================

expert advices are most welcomed in the meanwhile...


regards
yogeeraj