Replaced failed mirrored drive - still NO_HW in ioscan

[Michael Streifel]

I got the following error from EMS:
/storage/events/disks/default/0_0_2_0.0.0 is >= 3.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Mon Jul 18 12:35:14 2005
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 3
System..............: ustlogs

Summary:
Disk at hardware path 0/0/2/0.0.0 : Drive is not responding.

ioscan gives the following output:
scsi:wsio:T:T:F:31:188:131072:disk:sdisk:0/0/2/0.0.0:0 0 3 18 0 0 0 0 11 48 63 2
3 253 53 69 158 :1:root.sba.lba.c720.tgt.sdisk:sdisk:NO_HW:DEVICE:SEAGATE ST3920
4LC:2

This drive is part of a mirrored set that is the boot drive. I have replaced the disk but ioscan still states that there is NO_HW. If I try to look at disk devices in SAM it will hang. diskinfo -v /dev/rdsk/c2t0d0 returns diskinfo: can't SIOC_INQUIRY /dev/rdsk/c2t0d0: No such device or address. I have tried insf -e with no luck.

What am I missing?

[Rick Garland · Accepted solution]

Did you replace the drive?

Was it the same type of drive? LVD vs HVD?
FWSCSI?

HP will put some of their firmware on disks. Is it not a disk that came from HP?

[Uday_S_Ankolekar]

I guess you need to run replace_dsk command so ioscan can claim it.

/opt/fcms/bin/fcmsutil /dev/tdn replace_dsk nportid

nportid you will get it from syslog.log

-USA..

[Mel Burslan]

did you do the replacement yourself ? if yes where did the part come from ? HP or another source ? Was it hot swappable/pluggable ? If yes, try taking it out placing it back in again after spraying the contacts with canned air.

There is alwas a chance of getting hardware which is DOA. If I were you I would order another replacement right away, just in case.

[A. Clay Stephenson]

I assume you are NOT doing an ioscan -k but rather an ioscan -fn so that a new ioscan is actually done. The most likely explanation is that you have another failed disk - it happens. Check your cables and termination. It is very common for a failed terminator to cause this sort of problem. The SCSI bus must be terminated in EXACTLY two places -- on the physical ends of the bus. Generally the controller servers as one termination point. Also, at least one device on the bus must supply termination power.

An ioscan -fn posting (so that nothing is filtered out) might be of use.

[Michael Streifel]

Here is the full ioscan:
# ioscan -fn
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapte
r (582)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
lan 0 0/0/0/0 btlan CLAIMED INTERFACE HP PCI 10/100Base
-TX Core
/dev/diag/lan0 /dev/ether0 /dev/lan0
ext_bus 0 0/0/1/0 c720 CLAIMED INTERFACE SCSI C896 Ultra2
Wide LVD
target 0 0/0/1/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ext_bus 1 0/0/1/1 c720 CLAIMED INTERFACE SCSI C896 Ultra W
ide Single-Ended
target 1 0/0/1/1.0 tgt CLAIMED DEVICE
disk 0 0/0/1/1.0.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC

/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
target 2 0/0/1/1.7 tgt CLAIMED DEVICE
ctl 1 0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ext_bus 2 0/0/2/0 c720 CLAIMED INTERFACE SCSI C87x Ultra W
ide Single-Ended
target 3 0/0/2/0.0 tgt NO_HW DEVICE
disk 1 0/0/2/0.0.0 sdisk NO_HW DEVICE SEAGATE ST39204LC

/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
target 4 0/0/2/0.7 tgt CLAIMED DEVICE
ctl 2 0/0/2/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/0/2/1 c720 CLAIMED INTERFACE SCSI C87x Ultra W
ide Single-Ended
target 5 0/0/2/1.2 tgt CLAIMED DEVICE
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 3
04
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
target 6 0/0/2/1.7 tgt CLAIMED DEVICE
ctl 3 0/0/2/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
tty 0 0/0/4/0 asio0 CLAIMED INTERFACE PCI Serial (103c1
048)
/dev/GSPdiag1 /dev/mux0 /dev/tty0p1
/dev/diag/mux0 /dev/tty0p0 /dev/tty0p2
tty 1 0/0/5/0 asio0 CLAIMED INTERFACE PCI Serial (103c1
048)
/dev/GSPdiag2 /dev/mux1
/dev/diag/mux1 /dev/tty1p1
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ba 3 0/3 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ba 4 0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ext_bus 4 0/4/0/0 c720 CLAIMED INTERFACE SCSI C87x Fast Wi
de Differential
target 7 0/4/0/0.0 tgt CLAIMED DEVICE
disk 3 0/4/0/0.0.0 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 5 0/4/0/0.0.1 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
disk 6 0/4/0/0.0.2 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d2 /dev/rdsk/c4t0d2
disk 7 0/4/0/0.0.3 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d3 /dev/rdsk/c4t0d3
disk 8 0/4/0/0.0.4 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d4 /dev/rdsk/c4t0d4
disk 9 0/4/0/0.0.5 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t0d5 /dev/rdsk/c4t0d5
target 8 0/4/0/0.1 tgt CLAIMED DEVICE
disk 4 0/4/0/0.1.0 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d0 /dev/rdsk/c4t1d0
disk 10 0/4/0/0.1.1 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d1 /dev/rdsk/c4t1d1
disk 11 0/4/0/0.1.2 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d2 /dev/rdsk/c4t1d2
disk 12 0/4/0/0.1.3 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d3 /dev/rdsk/c4t1d3
disk 13 0/4/0/0.1.4 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d4 /dev/rdsk/c4t1d4
disk 14 0/4/0/0.1.5 sdisk CLAIMED DEVICE HP C5447A
/dev/dsk/c4t1d5 /dev/rdsk/c4t1d5
target 9 0/4/0/0.3 tgt CLAIMED DEVICE
tape 0 0/4/0/0.3.0 stape CLAIMED DEVICE QUANTUM DLT8000
/dev/rmt/0m /dev/rmt/c4t3d0BEST
/dev/rmt/0mb /dev/rmt/c4t3d0BESTb
/dev/rmt/0mn /dev/rmt/c4t3d0BESTn
/dev/rmt/0mnb /dev/rmt/c4t3d0BESTnb
target 10 0/4/0/0.7 tgt CLAIMED DEVICE
ctl 4 0/4/0/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
ba 5 0/5 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ba 6 0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
ba 7 0/7 lba CLAIMED BUS_NEXUS Local PCI Bus Ada
pter (782)
memory 0 8 memory CLAIMED MEMORY Memory
processor 0 160 processor CLAIMED PROCESSOR Processor
processor 1 166 processor CLAIMED PROCESSOR Processor

I have replaced the disk and it is the same part number that was removed (A5802A). I purchased the drive from an HP reseller and they tested it before sending it out. Both of the drives are a L2000 server, one is in slot A0 and the bad one is in slot B0. The both are hotswapable.

How do I determine the nportid from the syslog? Here is some of the errors in the syslog:
Jul 13 10:11:24 ustlogs vmunix: LVM: vg[0]: pvnum=0 (dev_t=0x1f020000) is POWERF
AILED
Jul 13 10:11:24 ustlogs vmunix: SCSI: Write error -- dev: b 31 0x020000, errno:
126, resid: 10240,
Jul 13 10:11:24 ustlogs vmunix: blkno: 2310, sectno: 4620, offset: 23654
40, bcount: 10240.
Jul 13 10:11:24 ustlogs vmunix:
Jul 13 10:21:51 ustlogs above message repeats 15 times

After the disk was replaced:
Jul 19 12:13:28 ustlogs vmunix: SCSI: Reset detected -- lbolt: 259779412, bus: 2
Jul 19 12:13:28 ustlogs vmunix: lbp->state: 6060
Jul 19 12:13:28 ustlogs vmunix: lbp->offset: ffffffff
Jul 19 12:13:28 ustlogs vmunix: lbp->uPhysScript: f8040000
Jul 19 12:13:28 ustlogs vmunix: From most recent interrupt:
Jul 19 12:13:28 ustlogs vmunix: ISTAT: 22, SIST0: 02, SIST1: 00,
DSTAT: 80, DSPS: 0000000a

[Mel Burslan]

Your disk which went out was a bad one for sure but the newly incoming one, despite being tested at the reseller, is not guaranteed to be good as it may have been damaged during transport.

Also, I came to crude realization that, in some weird cases, when you pull out a hot swappable SCSI drive, this may send SCSI bus to a sleep which it will refuse to wake up anything shourt of a total power-down and power-up. This happened to me on an rp8400 production server and we had to take the whole server down, power it down cold, wait for 30 seconds then bring it up. At which point all was cleared out.

Just a suggestion.

[Patrick Wallek]

On this disk, you don't need to worry about running the 'fcmsutil' command Uday gave. That is only for Fibre connected disks in something like an FC10.

I would try pulling the disk out, leave it out for a couple of minutes to let it spin down completely, and then re-insert it. If it still shows up as NO_HW, call the vendor back and tell them to ship you another disk as the one they originally sent is DOA.

[A. Clay Stephenson]

The most likely explanation is that the replacement disk is bad although you could have controller, cable, or termination problems but this is simple LVD SCSI so no Fibre is involved. I would knock this machine down and swap the drive locations and then do search for devices while under the firmware monitor. This should narrow down the problem to disk or other.

[Pradeep_29]

If u can reboot the machine, check at PDC prompt. Interrupt autoboot and run SEARCH

You should be able to see the disk. If not you can conclude with the disk connectivity or problem with disk itself. As someone suggested, check replacing the working disk in that shelf. I am sure you can understand which disk you r referring with hardware address.

If you are doing it online, I am not sure it will detect correctly. A simple solution would be to reboot. If you power on/off the disk array you can try this too. Be cautios about boot disk.

Thx,
Pradeep.

[Sudeesh]

Try doing
ioscan -H 0/0/2/0.0.0
This will try to reclaim the disk, then check with ioscan -fn

If the disk is still not claimed, you can go ahead with reboot.


Sudeesh

[Cheryl Griffin]

NO_HW is not good. It means there is still a hw problem, not a kernel driver problem, because tgt and sdisk are in the kernel (and as you can see, working for other disks).

Test the disk with dd, diskinfo and adb:
# dd if=/dev/dsk/cXdXtX of=/dev/null bs=64 count=1000
records in & out should match

# diskinfo /dev/rdsk/cxtxdx
The size should be correct, and not produce any syslog errors.

# echo 2400?20X | adb /dev/dsk/cxtxdx
The first two numbers should be the same, the rest are 0's.

[Andrew Rutter]

hi, from what I remember about the L class drives they are hotplugable(easy to change) but not hotswapable(while system is up)

You should power down the server, replace the drive and then power the system back up again.
check at pdc to check the drive is been seen ok, if not then the disk could be bad or the disk backplane could be bad in the server. Or blown if the drive was hotswapped and shorted out.
check its seen in another slot to verfiy the disk.

rebooting the server will probably fix this and cause the drive to spin up. It's maybe waiting for a signal from the scsi bus which will be issued upon power up.

Andy

[Michael Streifel]

That's for all your help. I was not on site and had someone else replace the disk. He didn't install the new drive all the way. I have reseated the drive and everything is as it should be.

Thanks again everyone.

[Devender Khatana]

Hi,

It was a quite long excercise to follow a human mistake. I was just wondering wheather you have mirrored it back.
What does

#lvdisplay -v /dev/vg00/lvol* shows.

HTH,
Devender