topic Re: 2 physical HDDs failed in Operating System - HP-UX

2 physical HDDs failed

Florian Dushniku — Tue, 10 May 2011 13:00:30 GMT

hi,
we run a rx 3600 hp ux server with 8 HDDs and RAID 5. recently 2 of HDDs failed. i want to know how to rebuild the RAID 5? where should I start? what should I do to get the server back toy work? I am sorry, i have very little experience with hp ux. i would be very thankful if you help me. thank you.

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 13:11:06 GMT

RAID 5 on internal disks?

So you have a P400 installed, right?

Look up the ioscan for the /dev/ciss entry and collect the current status with saconfig/sautil, e.g.

# saconfig /dev/ciss7

Re: 2 physical HDDs failed

Mike Miller_8 — Tue, 10 May 2011 13:14:21 GMT

This is every admin's nightmare. Unless you set up your raid array with 2 hot spares (probably not) you will probably need to rebuild it and recover your data from backups. Look for a doc called "when good disks go bad" (see attached). Good luck.

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 13:19:37 GMT

"when good disk go bad" will not help so much, because of hardware raid.

If you have at least 1 spare, even 2 failed disks should not cause data loss.

As said, you first need to get the current status from saconfig/sautil.

Re: 2 physical HDDs failed

Florian Dushniku — Tue, 10 May 2011 13:31:02 GMT

i run ioscan but i can't see /dev/ciss
# ioscan
H/W Path Class Description
=======================================================================
root
0 ioa System Bus Adapter (4030)
0/0 ba Local PCI-X Bus Adapter (122e)
0/0/1/0 tty PCI class(255,0) (103c1303)
0/0/1/1 tty PCI SimpleComm (103c1302)
0/0/1/2 tty PCI Serial (103c1048)
0/0/2/0 OO USB OHCI Interface
0/0/2/1 OO USB OHCI Interface
0/0/2/1.0 OO USB Composite Device
0/0/2/1.0.0 OO USB Bulk Only
0/0/2/1.0.16 ext_bus USB SCSI Stack Adaptor
0/0/2/1.0.16.0 target
0/0/2/1.0.16.0.0 disk TEAC DV-28E-N
0/0/2/1.0.16.7 target
0/0/2/1.0.16.7.0 ctl Initiator
0/0/2/2 OO USB EHCI Interface
0/1 ba Local PCI-X Bus Adapter (122e)
0/2 ba Local PCI-X Bus Adapter (122e)
0/3 ba Local Bus Adapter
0/3/0/0 ba PCItoPCI Bridge
0/4 ba Local PCI-X Bus Adapter (122e)
0/4/2/0 lan HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core
0/4/2/1 lan HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core
0/5 ba Local PCI-X Bus Adapter (122e)
0/6 ba Local Bus Adapter
0/6/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0/1/0 ba PCItoPCI Bridge
0/6/0/0/0/0/1/0/0/0 ext_bus PCIe SAS SmartArray P400 RAID Controller
0/6/0/0/0/0/1/0/0/0.0 target
0/6/0/0/0/0/1/0/0/0.0.0 disk HP LOGICAL VOLUME
0/6/0/0/0/0/1/0/0/0.0.1 disk HP LOGICAL VOLUME
0/6/0/0/0/0/1/0/0/0.7 target
0/6/0/0/0/0/1/0/0/0.7.0 ctl Initiator
0/7 ba Local Bus Adapter
0/7/0/0 ba PCItoPCI Bridge
120 processor Processor
121 processor Processor
250 ba Core I/O Adapter
250/0 ipmi IPMI Controller
250/1 tty Built-in RS232C
250/2 acpi_node Acpi Hardware

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 13:35:18 GMT

Use

# ioscan -fn

and check this line

0/6/0/0/0/0/1/0/0/0 ext_bus PCIe SAS SmartArray P400 RAID Controller

Re: 2 physical HDDs failed

Florian Dushniku — Tue, 10 May 2011 13:46:45 GMT

ok here is the output:
# saconfig /dev/ciss1

******************** SmartArray RAID Controller /dev/ciss1 ********************

Auto-Fail Missing Disks at Boot = disabled
Cache Configuration Status = cache enabled
Cache Ratio = 25% Read / 75% Write

---------- PHYSICAL DRIVES ----------

Location Ct Enc Bay WWID Size Status

Internal 1I 1 4 0x5000c500063292fd 146.8 GB OK
Internal 1I 1 3 0x5000c500063197a5 146.8 GB OK
Internal 1I 1 2 0x5000c50006327dd9 146.8 GB OK
Internal 1I 1 1 0x5000c50006321861 146.8 GB OK
Internal 2I 1 8 0x5000c500230cb4f1 146.8 GB OK
Internal 2I 1 7 0x5000c5000fbc64fd 146.8 GB OK
Internal 2I 1 6 0x5000c50006326d0d 146.8 GB OK
Internal 2I 1 5 0x5000c5000fbc5e5d 146.8 GB OK

---------- LOGICAL DRIVE 0 ----------

Device File = c1t0d0
RAID Level = 1+0
Size = 139979 MB
Stripe Size = 128 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1I 1 2 0x5000c50006327dd9
1I 1 1 0x5000c50006321861

Participating Spare Drive(s):

None

---------- LOGICAL DRIVE 1 ----------

Device File = c1t0d1
RAID Level = 5
Size = 699911 MB
Stripe Size = 16 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1I 1 4 0x5000c500063292fd
1I 1 3 0x5000c500063197a5
2I 1 8 0x5000c500230cb4f1
2I 1 7 0x5000c5000fbc64fd
2I 1 6 0x5000c50006326d0d
2I 1 5 0x5000c5000fbc5e5d

Participating Spare Drive(s):

None

#

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 13:50:18 GMT

According to this there is no problem at all!

All 8 disks and both logical drives are fine.

Why you think you have 2 failed drives?

Re: 2 physical HDDs failed

Florian Dushniku — Tue, 10 May 2011 14:11:43 GMT

some days ago 2 physical hdds in the slots 5 and 7 failed and i removed them. some days later i put in the 2 empty slots 2 new hdds. now to explain to you better the problem i am attaching some print screens that talk by themselves... as far as i understand there are 2 volume groups, but i can't mount /oracle, or activate it. when i access the folders of these server with winscp the oracle folder is empty. what can i do about this? thank you very much for helping me.

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 15:58:02 GMT

You just replaced the disks, no other actions taken?

Just try

# vgchange -a y oraclevg

# vgdisplay -v oraclevg

Re: 2 physical HDDs failed

Torsten. — Tue, 10 May 2011 16:09:38 GMT

... but if the logical drive was in status FAILED and you did just an enable, so the data is lost and you need to restore the LVM configuration and data.

Re: 2 physical HDDs failed

Florian Dushniku — Tue, 10 May 2011 18:48:33 GMT

yes, i just replaced the disks, no other action. the output from the commands, below:
# vgchange -a y oraclevg
vgchange: Couldn't activate volume group "oraclevg":
Quorum not present, or some physical volume(s) are missing.

# vgdisplay -v oraclevg
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "oraclevg".
#

how can i restore the LVM configuration and data?

Re: 2 physical HDDs failed

Torsten. — Wed, 11 May 2011 03:57:33 GMT

To restore the configuration please do

# vgcfgrestore -n oraclevg /dev/rdsk/c1t0d1

# vgchange -a y oraclevg

Then you need to create the filesystems and restore the data.

Re: 2 physical HDDs failed

Florian Dushniku — Wed, 11 May 2011 05:18:42 GMT

ok. the 2 commands above were successful. now, in the system management homepage it seems ok except the print screen i am attaching...so, i just need to mount /oracle? how can i create the filesystems and restore the data?; by just reinstalling oracle?

Re: 2 physical HDDs failed

g3jza — Wed, 11 May 2011 05:32:22 GMT

Hi,
you should know what was the layout of your FS / LVM before this accident happened to you.

You should see the /etc/fstab to see what LV was mounted on which mount point.

Then just create new FS on the corresponding lvol's:
#newfs -F vxfs /dev/vgora/rlvolX

and restore the data from recent backup. You are making backups, don't you?

Re: 2 physical HDDs failed

g3jza — Wed, 11 May 2011 05:36:07 GMT

so just newfs -F vxfs /dev/oraclevg/rorcl
in your case...

Re: 2 physical HDDs failed

Torsten. — Wed, 11 May 2011 05:36:18 GMT

vgdisplay -v oraclevg

will show you the lvols inside the vg, most likely it is "orcl" (according to the screenshot).

So you need to

# newfs -F vxfs /dev/oraclevg/rorcl

then mount it

# mount -a

and restore the data from backup.

Re: 2 physical HDDs failed

Florian Dushniku — Wed, 11 May 2011 05:59:58 GMT

thank you for the answers. the output from the commands are:
# vgdisplay -v oraclevg
--- Volume groups ---
VG Name /dev/oraclevg
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 43745
VGDA 2
PE Size (Mbytes) 16
Total PE 43738
Alloc PE 43738
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/oraclevg/orcl
LV Status available/syncd
LV Size (Mbytes) 699808
Current LE 43738
Allocated PE 43738
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c1t0d1
PV Status available
Total PE 43738
Free PE 0
Autoswitch On

# newfs -F vxfs /dev/oraclevg/rorcl
version 5 layout
716603392 sectors, 716603392 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles not supported
716603392 data blocks, 716406528 free data blocks
21869 allocation units of 32768 blocks, 32768 data blocks
# mount -a
vxfs mount: mount option(s) incompatible with file system /dev/oraclevg/orcl
mount: /dev/vg00/lvol8 is already mounted on /var
mount: /dev/vg00/lvol7 is already mounted on /usr
mount: /dev/vg00/lvol6 is already mounted on /opt
mount: /dev/vg00/lvol5 is already mounted on /home
mount: /dev/vg00/lvol4 is already mounted on /tmp
mount: /dev/vg00/lvol1 is already mounted on /stand
#
attached the result in SMH... it is recognized now but still not mounted.

Re: 2 physical HDDs failed

Torsten. — Wed, 11 May 2011 06:09:15 GMT

Please show the /etc/fstab file

Re: 2 physical HDDs failed

g3jza — Wed, 11 May 2011 06:10:53 GMT

Yep,
you probably need to create the FS again, now with largefiles option enabled:

Whats in /etc/fstab? Is there option "largefiles" to the /dev/oraclevg/orcl? If yes, then re-create the FS again:

newfs -F vxfs -o largefiles /dev/oraclevg/rorcl

Or if having a license for OnlineJFS, do it online without unmounting the FS:
fsadm -o largefiles /oracle