Disk Failure?

[Yogeeraj_1]

Dear experts!

I suspect a disk crash on my system! This is an internal disk on my rp5400. below the symptoms:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 2 0/0/1/1.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 3 0/0/1/1.2.0 sdisk NO_HW DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 4 0/0/2/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 0 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 1 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
#

Extract from my syslog file
===========================
Dec 17 11:06:21 l1000 vmunix: SCSI: Request Timeout -- lbolt: 332150, dev: 1f012000
Dec 17 11:06:21 l1000 vmunix: lbp->state: 4060
Dec 17 11:06:21 l1000 vmunix: lbp->offset: ffffffff
Dec 17 11:06:21 l1000 vmunix: lbp->uPhysScript: f87be000
Dec 17 11:06:21 l1000 vmunix: From most recent interrupt:
Dec 17 11:06:21 l1000 vmunix: ISTAT: 22, SIST0: 00, SIST1: 04, DSTAT:00, DSPS: f87be540
Dec 17 11:06:21 l1000 vmunix: lsp: 0000000049a34c00
Dec 17 11:06:21 l1000 vmunix: bp->b_dev: 1f012000
Dec 17 11:06:21 l1000 vmunix: scb->io_id: 10003e4
Dec 17 11:06:21 l1000 vmunix: scb->cdb: 28 00 01 ec ba f8 00 00 08 00
Dec 17 11:06:21 l1000 vmunix: lbolt_at_timeout: 329122, lbolt_at_start: 329122
Dec 17 11:06:21 l1000 vmunix: lsp->state: 10d
Dec 17 11:06:21 l1000 vmunix: lbp->owner: 0000000049a34c00
Dec 17 11:06:21 l1000 vmunix: scratch_lsp: 0000000000000000
Dec 17 11:06:21 l1000 vmunix: Pre-DSP script dump [fffffffff87be020]:
Dec 17 11:06:21 l1000 vmunix: 00000000 00000000 41020000 f87be290
Dec 17 11:06:21 l1000 vmunix: 78347b00 0000000a 78350800 00000000
Dec 17 11:06:21 l1000 vmunix: Script dump [fffffffff87be040]:
Dec 17 11:06:21 l1000 vmunix: 0e000004 f87be540 e0100004 f87be7ec
Dec 17 11:06:21 l1000 vmunix: 870b0000 f87be2d8 0a000000 f87be548
Dec 17 11:06:21 l1000 vmunix: SCSI: Abort abandoned -- lbolt: 332150, dev: 1f012000, io_id: 10003e4, status: 200
Dec 17 11:06:21 l1000 vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f012000) is POWERFAILED


Below Server MAp output from STM:
=================================
l1000

Dev Last Last Op
Num Path Product Active Tool Status
=== ==================== ========================= =========== =============
1 system system () Information Successful
2 0 Bus Adapter (582) Information Successful
3 0/0 PCI Bus Adapter (782) Information Successful
4 0/0/0/0 Core PCI 100BT Interface Information Successful
5 0/0/1/0 PCI SCSI Interface (10000 Verify Successful
6 0/0/1/1 PCI SCSI Interface (10000 Verify Successful
7 0/0/1/1.0.0 SCSI Disk (SEAGATEST31840 Verify Successful
8 0/0/1/1.2.0 Unknown ()
9 0/0/2/0 PCI SCSI Interface (10000 Verify Successful
10 0/0/2/0.0.0 SCSI Disk (SEAGATEST31840 Verify Successful
11 0/0/2/0.2.0 SCSI Disk (SEAGATEST31840 Verify Successful
12 0/0/2/1 PCI SCSI Interface (10000 Verify Successful
13 0/0/2/1.2.0 SCSI Disk (HPDVD-ROM) Information Successful
14 0/0/4/0 RS-232 Interface (103c104 Information Successful
15 0/0/5/0 RS-232 Interface (103c104 Information Successful
16 0/1 PCI Bus Adapter (782) Information Successful
17 0/2 PCI Bus Adapter (782) Information Successful
18 0/2/0/0 PCI Bus Adapter (10110024 Information Successful
19 0/2/0/0/4/0 PCI 4 Port 100BT LAN (101 Information Successful
20 0/2/0/0/5/0 PCI 4 Port 100BT LAN (101 Information Successful
21 0/2/0/0/6/0 PCI 4 Port 100BT LAN (101 Information Successful
22 0/2/0/0/7/0 PCI 4 Port 100BT LAN (101 Information Successful
23 0/3 PCI Bus Adapter (782) Information Successful
24 0/3/0/0 1000 Base-XX LAN Interfac Information Successful
25 0/4 PCI Bus Adapter (782) Information Successful
26 0/4/0/0 PCI SCSI Interface (10000 Verify Successful
27 0/4/0/1 PCI SCSI Interface (10000 Verify Successful
28 0/5 PCI Bus Adapter (782) Information Successful
29 0/6 PCI Bus Adapter (782) Information Successful
30 0/7 PCI Bus Adapter (782) Information Successful
31 0/7/0/0 PCI SCSI Interface (10000 Verify Successful
32 0/7/0/1 PCI SCSI Interface (10000 Verify Successful
33 8 MEMORY (95) Information Successful
34 160 CPU (5c9) Information Successful
35 es FC Network Storage (455)


diskinfo
========
# diskinfo /dev/rdsk/c1t0d0
SCSI describe of /dev/rdsk/c1t0d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c1t2d0
diskinfo: can't SIOC_INQUIRY /dev/rdsk/c1t2d0: No such device or address
# diskinfo /dev/rdsk/c2t0d0
SCSI describe of /dev/rdsk/c2t0d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c2t2d0
SCSI describe of /dev/rdsk/c2t2d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
#

dd
=====
# dd if=/dev/rdsk/c1t2d0 of=/dev/null bs=64k

(1 hours already elapse, still no response)

What should i do next to confirm if the hard disk has really crashed?

thanking you in advance for your replies

kind regards
yogeeraj

ps. (Attached document contains the same information as above -- which is more readable)

[RAC_1]

If disk has failed and not visible to OS, then dd or diskinfo are of no use. They just can reach the disk to do anything. The disk indeed seems to be failed.

If you get downtime, Check cables, disk seating and if required, replace it.

[Adisuria Wangsadinata_1]

Hi Yogeeraj,

The disk /dev/dsk/c1t2d0 is confirmed need to replace. Because :
- from ioscan, it's already with NO_HW status
- from syslog, pointing to dev: 1f012000 (which is c1t2d0)
- from STM, the device (HW path : 0/0/1/1.2.0) with 'unknown' status
- from diskinfo, you already got the error

for dd command is better to use bigger block size (eg. 1024k), but I bet the result will be the same (too long and you will got I/O error message).

I refer you to replace this device and prepare the backup.

Hope this information can help you.

Cheers,
AW

[Arunvijai_4]

Hi Yogee, Can you please check this thread, this may be helpful to you.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=941125

-Arun

[Yogeeraj_1]

Dear all,

thank you for the replies.

Good heaven! do we call this bad luck?
2 disk crashes in a month! Once of my AutoRAID disk and now this one!

Fortunately, these are part of my backup infrastructure!

Can anyone advise me on the replacement disk that i can purchase now? Need one with higher capacity of course...


thank you again for confirming

kind regards
yogeeraj

[Adisuria Wangsadinata_1 · Accepted solution]

Hi Yogeeraj,

If you want to replace the disk with the higher capacity, you need to check the disks configuration on your system. Whether a mirrored OR a non mirrored configuration; since if mirrored, you need to purchase not only one since you also have a mirrored part that need to be extend also (so both side will be equal).

From the PartSurfer, the disk for rp5400 are 9GB, 18GB, 36GB :

http://partsurfer.hp.com/cgi-bin/spi/main?sel_flg=partlist&model=L1000&HP_model=&modname=9000+Server+-+L1000%2FB&template=main&plist_sval=Drive&plist_styp=subcat&dealer_id=&callingsite=&strsrch=&keysel=Drive&catsel=%3F

Check the url below for your refrence for 'Dealing with Disk Failures under LVM' :

http://www.docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

Hope this information can help you.

Cheers,
AW

[Cem Tugrul]

Yogeeraj,
if your disks really crashed then you
could not detect from Os also diskinfo
as Adisuria's mentioned but at the end you seem to change these disks but before this
if they are hotplug(propably) then just
pull out and then again push in the disk...Once upon a time i saved a disk from my FC60 array like this and it is still working.

Good Luck,

[Robert-Jan Goossens]

Yogeeraj,

There is also a 146GB drive available for the rp4500.

36 GB, 15K HotPlug Ultra 160 SCSI Disk A6846A
73 GB, 15K HotPlug Ultra 160 SCSI Disk A9761A
146 GB, 10K HotPlug Ultra 160 SCSI Disk A7080A

http://h10025.www1.hp.com/ewfrf/wc/genericDocument?cc=us&docname=c00288650&lc=en#drive

Regards,
Robert-Jan

[Yogeeraj_1]

hi again,

thank you for the replies
In fact, my server is an L1000. I guess the same part numbers are applicable. right?

thanks
yogeeraj

[Arunvijai_4]

http://partsurfer.hp.com/cgi-bin/spi/main?sel_flg=partlist&model=L1000&HP_model=&modname=9000+Server+-+L1000%2FB&template=main&plist_sval=Drive&plist_styp=subcat&dealer_id=&callingsite=&strsrch=&keysel=Drive&catsel=%3F

[Arunvijai_4]

There is 73GB 10k disk available from partsurfer, http://partsurfer.hp.com/cgi-bin/spi/main?sel_flg=partlist&model=L1000&HP_model=&modname=9000+Server+-+L1000%2FB&template=main&plist_sval=Drive+%28Product%29&plist_styp=subcat&dealer_id=&callingsite=&strsrch=&keysel=Drive+%28Product%29&catsel=%3F

-Arun

[Yogeeraj_1]

hi again!
==============================================
Part Number Price Part Description
A5803-69003 435.00 Drive : 18GB hot-plug Ultra-2 SCSI hard drive module - 10,000 RPM, low profile (Seagate ST318404LC)
REQUIRES RETURN OF DEFECTIVE PART

A5803-67002 967.00 New part for A5803-69003
==============================================

"REQUIRES RETURN OF DEFECTIVE PART"
Is this still applicable if:
1. my server is no longer under warranty/maintenance contract?
2. I want to upgrade to a higher capacity?


please clarify...

kind regards
yogeeraj

[Arunvijai_4]

Hi Yogee, This is what i can see from Userguide about RETURN OF DEFECTIVE PART.

"The parts listings also include references between exchange and new part number: Exchange parts are refurbished parts available at a lower price with the return to HP of the old part. Exchange parts are indicated in the parts listings by the note REQUIRES RETURN OF DEFECTIVE PART in the description. The corresponding new part number is displayed immediately following, with it�s information in italics."

Please call HP Customer care to verify whether you have to return the to get a new higher capacity one.

http://partsurfer.hp.com/userguide.htm

-Arun

[Yogeeraj_1]

hi again!

!!!!New information!!!!
We have reseated the disk and below the output we are getting.

============================================================================
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 2 0/0/1/1.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 3 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 4 0/0/2/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 0 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
#

Extract from my syslog file
===========================
Dec 21 15:30:03 l1000 vmunix: SCSI: Unhandled interrupt -- lbolt: 6187, dev: cb050002
Dec 21 15:30:03 l1000 vmunix: lbp->state: 60
Dec 21 15:30:03 l1000 vmunix: lbp->offset: ffffffff
Dec 21 15:30:03 l1000 vmunix: lbp->uPhysScript: fa7df000
Dec 21 15:30:03 l1000 vmunix: From most recent interrupt:
Dec 21 15:30:03 l1000 vmunix: ISTAT: 0a, SIST0: 91, SIST1: 00, DSTAT:
80, DSPS: 00138300
Dec 21 15:30:03 l1000 vmunix: lsp: 0000000041391200
Dec 21 15:30:03 l1000 vmunix: bp->b_dev: cb050002
Dec 21 15:30:03 l1000 vmunix: scb->io_id: 5000001
Dec 21 15:30:03 l1000 vmunix: scb->cdb: 12 00 00 00 80 00
Dec 21 15:30:03 l1000 vmunix: lbolt_at_timeout: 0, lbolt_at_start: 0
Dec 21 15:30:03 l1000 vmunix: lsp->state: 5
Dec 21 15:30:03 l1000 vmunix: lbp->owner: 0000000000000000
Dec 21 15:30:03 l1000 vmunix: scratch_lsp: 0000000041391200
Dec 21 15:30:03 l1000 vmunix: Script dump [0000000044010000]:
Dec 21 15:30:03 l1000 vmunix: 09000080 00138300 e25c0004 fa7df7f8
Dec 21 15:30:03 l1000 vmunix: 80080000 fa7df090 80080000 f87ba090
Dec 21 15:30:03 l1000 vmunix: SCSI: Resetting SCSI -- lbolt: 6287, bus: 5
Dec 21 15:30:03 l1000 vmunix: SCSI: Reset detected -- lbolt: 6287, bus: 5


Below Server MAp output from STM:
=================================
l1000

Dev Last Last Op
Num Path Product Active Tool Status
=== ==================== ========================= =========== =============
1 system system () Information Successful
2 0 Bus Adapter (582) Information Successful
3 0/0 PCI Bus Adapter (782) Information Successful
4 0/0/0/0 Core PCI 100BT Interface Information Successful
5 0/0/1/0 PCI SCSI Interface (10000 Verify Successful
6 0/0/1/1 PCI SCSI Interface (10000 Verify Successful
7 0/0/1/1.0.0 SCSI Disk (SEAGATEST31840 Information Successful
* 8 0/0/1/1.2.0 SCSI Disk (SEAGATEST31840 Information Successful
9 0/0/2/0 PCI SCSI Interface (10000 Verify Successful
10 0/0/2/0.0.0 SCSI Disk (SEAGATEST31840 Information Successful
11 0/0/2/0.2.0 SCSI Disk (SEAGATEST31840 Information Successful
12 0/0/2/1 PCI SCSI Interface (10000 Verify Successful
13 0/0/2/1.2.0 SCSI Disk (HPDVD-ROM) Information Successful
14 0/0/4/0 RS-232 Interface (103c104 Information Successful
15 0/0/5/0 RS-232 Interface (103c104 Information Successful
16 0/1 PCI Bus Adapter (782) Information Successful
17 0/2 PCI Bus Adapter (782) Information Successful
18 0/2/0/0 PCI Bus Adapter (10110024 Information Successful
19 0/2/0/0/4/0 PCI 4 Port 100BT LAN (101 Information Successful
20 0/2/0/0/5/0 PCI 4 Port 100BT LAN (101 Information Successful
21 0/2/0/0/6/0 PCI 4 Port 100BT LAN (101 Information Successful
22 0/2/0/0/7/0 PCI 4 Port 100BT LAN (101 Information Successful
23 0/3 PCI Bus Adapter (782) Information Successful
24 0/3/0/0 1000 Base-XX LAN Interfac Information Successful
25 0/4 PCI Bus Adapter (782) Information Successful
26 0/4/0/0 PCI SCSI Interface (10000 Verify Successful
27 0/4/0/1 PCI SCSI Interface (10000 Verify Successful
28 0/4/0/1.0 Disk Array (HPC3586A)
29 0/5 PCI Bus Adapter (782) Information Successful
30 0/6 PCI Bus Adapter (782) Information Successful
31 0/7 PCI Bus Adapter (782) Information Successful
32 0/7/0/0 PCI SCSI Interface (10000 Verify Successful
33 0/7/0/1 PCI SCSI Interface (10000 Verify Successful
34 0/7/0/1.1 Disk Array (HPC3586A)
35 8 MEMORY (95) Information Successful
36 160 CPU (5c9) Information Successful
37 es FC Network Storage (455)


diskinfo
========
# diskinfo /dev/rdsk/c1t0d0
SCSI describe of /dev/rdsk/c1t0d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c1t2d0
SCSI describe of /dev/rdsk/c1t2d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c2t0d0
SCSI describe of /dev/rdsk/c2t0d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
# diskinfo /dev/rdsk/c2t2d0
SCSI describe of /dev/rdsk/c2t2d0:
vendor: SEAGATE
product id: ST318404LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512
#
============================================================================

You will observe that it is now CLAIMing the disk that was not there in my initial post!

Also, the error message from the syslog file...


please advise on this

thanking you in advance for a reply!

kind regards
yogeeraj

[Yogeeraj_1]

hi again!

after about an hour, this what the system is showing me:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 2 0/0/1/1.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
disk 3 0/0/1/1.2.0 sdisk NO_HW DEVICE SEAGATE ST318404LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 4 0/0/2/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 0 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 1 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 304
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0


and the output of the dmesg command:
=====================================

SCSI: First party detected bus hang -- lbolt: 486993, bus: 1
lbp->state: 5060
lbp->offset: f0
lbp->uPhysScript: f87be000
From most recent interrupt:
ISTAT: 01, SIST0: 00, SIST1: 00, DSTAT: 84, DSPS: 00000010
lsp: 0000000000000000
lbp->owner: 0000000049a34c00
bp->b_dev: cb012000
scb->io_id: 1006539
scb->cdb: 12 00 00 00 80 00
lbolt_at_timeout: 486893, lbolt_at_start: 486393
lsp->state: 5
scratch_lsp: 0000000049a34c00
Pre-DSP script dump [fffffffff87be020]:
00000000 00000000 41020000 f87be290
78347d00 0000000a 78350800 00000000
Script dump [fffffffff87be040]:
0e000007 f87be540 e0100004 f87be7f4
870b0000 f87be2d8 98080000 00000005
SCSI: Resetting SCSI -- lbolt: 487593, bus: 1
SCSI: Reset detected -- lbolt: 487593, bus: 1


please advise
kind regards
yogeeraj

[Global Service Desk]

HI,

Its surely a DISK Problem. the time has come to replace it. this problem occurs when the disk failed.

[Eric Antunes]

Hi Yogeeraj,

Can you post the following pvdisplay.log resulting file:

#pvdisplay -v /dev/dsk/c1t2d0 > pvdisplay.log

Best Regards,

Eric Antunes

[Yogeeraj_1]

Dear Eric,

THank you for your reply,

attached the file that you have requested.

kind regards
yogeeraj

[Eric Antunes]

Hi Yogeeraj,

Your disk seems definitively KO.

Please post the following result to check if it is part of the boot path?

#setboot

Best Regards,

Eric Antunes

[Yogeeraj_1]

hi again!

thank you again for your reply.

Below the information that you requested:
# setboot
Primary bootpath : 0/0/2/0.2.0
Alternate bootpath : 0/0/2/0.2.0

Autoboot is ON (enabled)
Autosearch is ON (enabled)

#

fortunately, i only have to creat the swap and lv_02 and lv_u03.

kind regards
yogeeraj

[Eric Antunes]

Hi again Yogeeraj,

I recomend you strongly to change your boot configuration because if /dev/dsk/c2t0d0 (path 0/0/2/0.0.0) fails YOU WONT BE ABLE TO BOOT!

You can define an alternate boot path with "setboot -a". I think 0/0/2/0.2.0 path - disk dev/dsk/c2t2d0 - is the logical choice because it seems to be part of the same volume group.

Execute "vgdisplay" to confirm that both /dev/dsk/c2t0d0 and /dev/dsk/c2t2d0 disks belongs to the same volume group:

#vgdisplay -v vg00|more
(this supposes the disks belongs to vg00. If they don't try with vg01, vg02, ... until you find them both)

So, in the "vgdisplay" result, both disks must be listed in the " --- Physical volumes --- " part, at the end.

If they are, then the "setboot" command:

#setboot -a 0/0/2/0.2.0

Best Regards,

Eric Antunes

[Yogeeraj_1]

hi Eric,

thank you for your your guidance. Unfortunately, the disks are not mirrored. AS mentioned earlier, the server is used as a backup one.

below, the outputs of vgdisplay commands:
# vgdisplay -v vg00|tail -8
--- Physical volumes ---
PV Name /dev/dsk/c2t2d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On


# vgdisplay -v vg01|tail -8

PV Name /dev/dsk/c2t0d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On


# vgdisplay -v vg02|tail -8
--- Physical volumes ---
PV Name /dev/dsk/c1t2d0
PV Status available
Total PE 4340
Free PE 0
Autoswitch On


i believe i cannot run the setboot command on any other hard drives...

kind regards
yogeeraj

[Eric Antunes]

Hi Yogeeraj,

I think you should try "setboot -a 0/0/2/0.2.0". I don't see any mirrorring requirements in the setboot manual.

Basically, setting an alternate path will enable you to boot even if the primary path has failed.

Best Regards,

Eric Antunes