HP StorageWorks 30 LOGICAL DRIVE 0 FAILED

[Timothy Nibbe]

I have an rp3400 running HPUX 11.11 with an attached HP StorageWorks 30 RAID array. The array is one logical drive with 7 + 1 physical drives.

One of the drives failed over the weekend and the filesystems on the array failed. Today, the bad disk was hot swapped. After the hot swap the array did not recover and after about two hours I rebooted the server.

The array still didn't come back up.

Can somebody please point out what I might be doing wrong? If the data is gone, is there a procedure for rebuilding the array? I have current backups.


Some outputs:

abhpux01:/ # ioscan -kfn | grep ciss
ext_bus 4 0/4/1/0/4/0 ciss CLAIMED INTERFACE PCI-X SmartArray 6402 RAID Controller
/dev/ciss4

abhpux01:/ # ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 36.4GST336753LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336754LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 3 0/4/1/0/4/0.0.0 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0


abhpux01:/ # /opt/raidsa/bin/saconfig /dev/ciss4

******************** SmartArray RAID Controller /dev/ciss4 ********************

Auto-Fail Missing Disks at Boot = enabled
Cache Configuration Status = write cache temporarily disabled (code=1)
Cache Ratio = 25% Read / 75% Write

---------- PHYSICAL DRIVES ----------

Location Ch ID Size Status

External 2 0 72.8 GB OK
External 2 1 72.8 GB OK
External 2 2 72.8 GB OK
External 2 3 72.8 GB OK
External 2 4 72.8 GB OK
External 2 5 72.8 GB OK
External 2 8 72.8 GB OK
External 2 9 72.8 GB SPARE

---------- LOGICAL DRIVE 0 ----------

RAID Level = 5
Size = 416761 MB
Stripe Size = 16 KB
Status = FAILED

Participating Physical Drive(s):

Ch ID

2 0
2 1
2 2
2 3
2 4
2 5
2 8

Participating Spare Drive(s):

Ch ID

2 9



abhpux01:/ # pvdisplay /dev/dsk/c4t0d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c4t0d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg01".
pvdisplay: Cannot display physical volume "/dev/dsk/c4t0d0".


abhpux01:/ # sautil /dev/ciss4

******************************************************************************
**** ****
**** S A U T I L S u p p o r t U t i l i t y ****
**** ****
**** for the HP SmartArray RAID Controller Family ****
**** ****
**** version A.02.06 ****
**** ****
**** (C) Copyright 2003, 2004 Hewlett-Packard Development Company, L.P. ****
******************************************************************************



---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... A9890A
Controller Product Name............. HP PCI-X SmartArray 6402
Hardware Path....................... 0/4/1/0/4/0
Serial Number....................... P57820PXQRQ3L2
Device File......................... /dev/ciss4
Hardware Revision................... 'B'
Boot Block Revision................. 3.86
Firmware Revision (running)......... 2.34
Firmware Revision (in ROM).......... 2.34
Firmware Revision (inactive)........ 2.34
# of Logical Drives................. 1
# of Physical Disks Configured...... 8
# of Physical Disks Detected........ 8
Logical Drive Rebuild Priority...... 118 (high)
Array Capacity Expansion Priority... 0 (low)
Auto-Fail Missing Disks at Boot..... enabled
SCSI Transfer Detection Mode........ Auto Detect


---- ARRAY ACCELERATOR (CACHE) INFORMATION -----------------------------------

Array Accelerator Board Present?.... yes
Cache Configuration Status.......... write cache temporarily disabled (code=1)
Cache Ratio......................... 25% Read / 75% Write
Total Cache Size (MB)............... 128
Read Cache........................ 032
Write Cache....................... 096
Transfer Buffer................... 000
Battery Pack Count.................. 2
Battery Status (pack #1)............ FAILED
Battery Status (pack #2)............ ok


---- LOGICAL DRIVE SUMMARY ---------------------------------------------------

# RAID Size Status

0 5 416761 MB FAILED


---- SCSI DEVICE SUMMARY -----------------------------------------------------

Location Ch ID Type Capacity Status

external 2 0 DISK 72.8 GB OK
external 2 1 DISK 72.8 GB OK
external 2 2 DISK 72.8 GB OK
external 2 3 DISK 72.8 GB OK
external 2 4 DISK 72.8 GB OK
external 2 5 DISK 72.8 GB OK
external 2 7 PROCESSOR
external 2 8 DISK 72.8 GB OK
external 2 9 DISK 72.8 GB SPARE


---- SCSI BUS SUMMARY --------------------------------------------------------

External Internal
Ch LVD_Mode Connector Connector

1 YES - -
2 YES YES -


---- LOGICAL DRIVE 0 ---------------------------------------------------------

Fault Tolerance Mode................ RAID 5 (Distributed Data Guarding)
Logical Drive Size.................. 416761 MB
Logical Drive Status................ FAILED
# of Participating Physical Disks... 7

Participating Physical Disk(s)...... Ch:ID
2: 0
2: 1
2: 2
2: 3
2: 4
2: 5
2: 8

Participating Spare Disk(s)......... Ch:ID
2: 9

Stripe Size......................... 16 KB
Logical Drive Cache Status.......... write cache temporarily disabled (code=1)
Configuration Signature............. 0xA2F924C7
Media Exchange Detected?............ yes


---- SCSI DEVICE 2:0 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 0
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288285
Device Serial Number............................. 3KP0M37M00007546VR15
Device Firmware Version.......................... HPB3
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. REMOVED IN HOT PLUG

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:1 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 1
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ASA1P5600A470526
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:2 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 2
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ANA1P5705HA40527
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:3 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 3
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ASA1P5600A4R0526
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:4 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 4
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ANA1P5705GM80527
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:5 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 5
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ASA1P5600A4T0526
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. none

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:7 [PROCESSOR] ---------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 7
Device Type...................................... PROCESSOR
Device Vendor ID................................. COMPAQ
Device Product ID................................ PROLIANT 4LEE
SCSI Transfer Rate (maximum)..................... Ultra-320


---- SCSI DEVICE 2:8 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 8
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288576
Device Serial Number............................. ANA1P5705GMN0527
Device Firmware Version.......................... HPB2
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. TIMEOUT

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... no
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- SCSI DEVICE 2:9 [DISK] --------------------------------------------------

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 9
Device Type...................................... DISK
Disk Capacity.................................... 72.8 GB
Device Status.................................... SPARE
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF07288285
Device Serial Number............................. 3KP0MP3X000075475G40
Device Firmware Version.......................... HPB3
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. TIMEOUT

Physical Disk Flags:
Disk present and operational................... yes
Non-disk device detected....................... no
Wide SCSI transfer enabled..................... yes
Synchronous (Fast/Ultra) transfer enabled...... yes
Narrow disk tray detected...................... no
Wide transfer failed, reverted to narrow....... no
Ultra SCSI transfer enabled.................... yes
Ultra-2 SCSI transfer enabled.................. yes

S.M.A.R.T. supported........................... yes
S.M.A.R.T. errors (in factory M&P data)........ no
S.M.A.R.T. enabled............................. yes
S.M.A.R.T. errors (in powerup M&P data)........ no
Attached to external channel................... yes
Configured in a logical drive.................. yes
Configured as a spare disk..................... yes
Disk write cache enabled at spin up............ no

Running in DT clocking mode.................... yes
Supports DT clocking mode only................. no
Supports DT and ST clocking modes.............. yes
Running in U320 mode........................... yes
Packetized SCSI supported...................... yes
Supports redundant controller operation........ yes
Disk write cache enabled in current page....... no
Disk write cache disabled in default page...... yes

QAS capable (U320 only)........................ yes
Currently running QAS (U320 only).............. yes



---- DIAGRAM OF EXTERNAL CONNECTORS ON BRACKET -------------------------------

[rest snipped]

[Torsten. · Accepted solution]

I wonder how this could happen.

Maybe the accept_media_xchg command can help.

Example:

# sautil /dev/ciss4 accept_media_xchg 0

see

http://docs.hp.com/en/J6369-90051/ch04s04.html#bcgggghf

[skris]

Yep,
Spot on Torsten

# sautil /dev/ciss4 accept_media_xchg 0

This forcefully re-enables the volumes on the MSA. Also one of the battery pack seems to have failed and needs to be replaced.

I found an available update to the SA 6402 Firmware (2.84) too, which you could consider too.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=378439&prodTypeId=329290&prodSeriesId=1132643&swLang=8&taskId=135&swEnvOID=54

[Timothy Nibbe]

Thank you very much, the

# sautil /dev/ciss4 accept_media_xchg 0

command took care of the RAID problem.

Apparently either I messed up the PV yesterday or something else did. Even so, rerecreating the PV and five LVs is not bad at all.

I thank you again,

Tim

[Timothy Nibbe]

I realized that the data corruption had wiped the lvm headers, that is why the lvm data had to be rebuilt for this array (PV).

I lucked out that it happened over a weekend and there was little usage. The backup system had done a full backup within hours of the failure -- and the restore from tape actually worked.

[Timothy Nibbe]

Thanks again for the help.