2 physical HDDs failed

[Florian Dushniku]

hi,
we run a rx 3600 hp ux server with 8 HDDs and RAID 5. recently 2 of HDDs failed. i want to know how to rebuild the RAID 5? where should I start? what should I do to get the server back toy work? I am sorry, i have very little experience with hp ux. i would be very thankful if you help me. thank you.

[Torsten.]

RAID 5 on internal disks?

So you have a P400 installed, right?

Look up the ioscan for the /dev/ciss entry and collect the current status with saconfig/sautil, e.g.

# saconfig /dev/ciss7

[Mike Miller_8]

This is every admin's nightmare. Unless you set up your raid array with 2 hot spares (probably not) you will probably need to rebuild it and recover your data from backups. Look for a doc called "when good disks go bad" (see attached). Good luck.

[Torsten.]

"when good disk go bad" will not help so much, because of hardware raid.

If you have at least 1 spare, even 2 failed disks should not cause data loss.

As said, you first need to get the current status from saconfig/sautil.

[Florian Dushniku]

i run ioscan but i can't see /dev/ciss
# ioscan
H/W Path Class Description
=======================================================================
root
0 ioa System Bus Adapter (4030)
0/0 ba Local PCI-X Bus Adapter (122e)
0/0/1/0 tty PCI class(255,0) (103c1303)
0/0/1/1 tty PCI SimpleComm (103c1302)
0/0/1/2 tty PCI Serial (103c1048)
0/0/2/0 OO USB OHCI Interface
0/0/2/1 OO USB OHCI Interface
0/0/2/1.0 OO USB Composite Device
0/0/2/1.0.0 OO USB Bulk Only
0/0/2/1.0.16 ext_bus USB SCSI Stack Adaptor
0/0/2/1.0.16.0 target
0/0/2/1.0.16.0.0 disk TEAC DV-28E-N
0/0/2/1.0.16.7 target
0/0/2/1.0.16.7.0 ctl Initiator
0/0/2/2 OO USB EHCI Interface
0/1 ba Local PCI-X Bus Adapter (122e)
0/2 ba Local PCI-X Bus Adapter (122e)
0/3 ba Local Bus Adapter
0/3/0/0 ba PCItoPCI Bridge
0/4 ba Local PCI-X Bus Adapter (122e)
0/4/2/0 lan HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core
0/4/2/1 lan HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core
0/5 ba Local PCI-X Bus Adapter (122e)
0/6 ba Local Bus Adapter
0/6/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0/0/0 ba PCItoPCI Bridge
0/6/0/0/0/0/1/0 ba PCItoPCI Bridge
0/6/0/0/0/0/1/0/0/0 ext_bus PCIe SAS SmartArray P400 RAID Controller
0/6/0/0/0/0/1/0/0/0.0 target
0/6/0/0/0/0/1/0/0/0.0.0 disk HP LOGICAL VOLUME
0/6/0/0/0/0/1/0/0/0.0.1 disk HP LOGICAL VOLUME
0/6/0/0/0/0/1/0/0/0.7 target
0/6/0/0/0/0/1/0/0/0.7.0 ctl Initiator
0/7 ba Local Bus Adapter
0/7/0/0 ba PCItoPCI Bridge
120 processor Processor
121 processor Processor
250 ba Core I/O Adapter
250/0 ipmi IPMI Controller
250/1 tty Built-in RS232C
250/2 acpi_node Acpi Hardware

[Torsten.]

Use

# ioscan -fn

and check this line

0/6/0/0/0/0/1/0/0/0 ext_bus PCIe SAS SmartArray P400 RAID Controller

[Florian Dushniku]

ok here is the output:
# saconfig /dev/ciss1

******************** SmartArray RAID Controller /dev/ciss1 ********************

Auto-Fail Missing Disks at Boot = disabled
Cache Configuration Status = cache enabled
Cache Ratio = 25% Read / 75% Write

---------- PHYSICAL DRIVES ----------

Location Ct Enc Bay WWID Size Status

Internal 1I 1 4 0x5000c500063292fd 146.8 GB OK
Internal 1I 1 3 0x5000c500063197a5 146.8 GB OK
Internal 1I 1 2 0x5000c50006327dd9 146.8 GB OK
Internal 1I 1 1 0x5000c50006321861 146.8 GB OK
Internal 2I 1 8 0x5000c500230cb4f1 146.8 GB OK
Internal 2I 1 7 0x5000c5000fbc64fd 146.8 GB OK
Internal 2I 1 6 0x5000c50006326d0d 146.8 GB OK
Internal 2I 1 5 0x5000c5000fbc5e5d 146.8 GB OK

---------- LOGICAL DRIVE 0 ----------

Device File = c1t0d0
RAID Level = 1+0
Size = 139979 MB
Stripe Size = 128 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1I 1 2 0x5000c50006327dd9
1I 1 1 0x5000c50006321861

Participating Spare Drive(s):

None

---------- LOGICAL DRIVE 1 ----------

Device File = c1t0d1
RAID Level = 5
Size = 699911 MB
Stripe Size = 16 KB
Status = OK

Participating Physical Drive(s):

Ct Enc Bay WWID
1I 1 4 0x5000c500063292fd
1I 1 3 0x5000c500063197a5
2I 1 8 0x5000c500230cb4f1
2I 1 7 0x5000c5000fbc64fd
2I 1 6 0x5000c50006326d0d
2I 1 5 0x5000c5000fbc5e5d

Participating Spare Drive(s):

None

#

[Torsten.]

According to this there is no problem at all!

All 8 disks and both logical drives are fine.


Why you think you have 2 failed drives?

[Florian Dushniku]

some days ago 2 physical hdds in the slots 5 and 7 failed and i removed them. some days later i put in the 2 empty slots 2 new hdds. now to explain to you better the problem i am attaching some print screens that talk by themselves... as far as i understand there are 2 volume groups, but i can't mount /oracle, or activate it. when i access the folders of these server with winscp the oracle folder is empty. what can i do about this? thank you very much for helping me.

[Torsten.]

You just replaced the disks, no other actions taken?

Just try

# vgchange -a y oraclevg

# vgdisplay -v oraclevg

[Torsten.]

... but if the logical drive was in status FAILED and you did just an enable, so the data is lost and you need to restore the LVM configuration and data.

[Florian Dushniku]

yes, i just replaced the disks, no other action. the output from the commands, below:
# vgchange -a y oraclevg
vgchange: Couldn't activate volume group "oraclevg":
Quorum not present, or some physical volume(s) are missing.

# vgdisplay -v oraclevg
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "oraclevg".
#

how can i restore the LVM configuration and data?

[Torsten.]

To restore the configuration please do

# vgcfgrestore -n oraclevg /dev/rdsk/c1t0d1

# vgchange -a y oraclevg

Then you need to create the filesystems and restore the data.

[Florian Dushniku]

ok. the 2 commands above were successful. now, in the system management homepage it seems ok except the print screen i am attaching...so, i just need to mount /oracle? how can i create the filesystems and restore the data?; by just reinstalling oracle?

[g3jza]

Hi,
you should know what was the layout of your FS / LVM before this accident happened to you.

You should see the /etc/fstab to see what LV was mounted on which mount point.

Then just create new FS on the corresponding lvol's:
#newfs -F vxfs /dev/vgora/rlvolX

and restore the data from recent backup. You are making backups, don't you?

[g3jza]

so just newfs -F vxfs /dev/oraclevg/rorcl
in your case...

[Torsten.]

vgdisplay -v oraclevg

will show you the lvols inside the vg, most likely it is "orcl" (according to the screenshot).

So you need to

# newfs -F vxfs /dev/oraclevg/rorcl

then mount it

# mount -a

and restore the data from backup.

[Florian Dushniku]

thank you for the answers. the output from the commands are:
# vgdisplay -v oraclevg
--- Volume groups ---
VG Name /dev/oraclevg
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 43745
VGDA 2
PE Size (Mbytes) 16
Total PE 43738
Alloc PE 43738
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/oraclevg/orcl
LV Status available/syncd
LV Size (Mbytes) 699808
Current LE 43738
Allocated PE 43738
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c1t0d1
PV Status available
Total PE 43738
Free PE 0
Autoswitch On

# newfs -F vxfs /dev/oraclevg/rorcl
version 5 layout
716603392 sectors, 716603392 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles not supported
716603392 data blocks, 716406528 free data blocks
21869 allocation units of 32768 blocks, 32768 data blocks
# mount -a
vxfs mount: mount option(s) incompatible with file system /dev/oraclevg/orcl
mount: /dev/vg00/lvol8 is already mounted on /var
mount: /dev/vg00/lvol7 is already mounted on /usr
mount: /dev/vg00/lvol6 is already mounted on /opt
mount: /dev/vg00/lvol5 is already mounted on /home
mount: /dev/vg00/lvol4 is already mounted on /tmp
mount: /dev/vg00/lvol1 is already mounted on /stand
#
attached the result in SMH... it is recognized now but still not mounted.

[Torsten.]

Please show the /etc/fstab file

[g3jza]

Yep,
you probably need to create the FS again, now with largefiles option enabled:

Whats in /etc/fstab? Is there option "largefiles" to the /dev/oraclevg/orcl? If yes, then re-create the FS again:

newfs -F vxfs -o largefiles /dev/oraclevg/rorcl

Or if having a license for OnlineJFS, do it online without unmounting the FS:
fsadm -o largefiles /oracle

[g3jza]

forget about the 'fsadm' thing....

[Florian Dushniku]

i used the following command. is it right?

# cat /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand vxfs tranflush 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2
/dev/oraclevg/orcl /oracle vxfs rw,suid,largefiles,delaylog,datainlog 0 2

[g3jza · Accepted solution]

ok,
recreate the FS:

#newfs -F vxfs -o largefiles /dev/oraclevg/rorcl

and then #mount /oracle

[Florian Dushniku]

ok. the output below:

# newfs -F vxfs -o largefiles /dev/oraclevg/rorcl
version 5 layout
716603392 sectors, 716603392 blocks of size 1024, log size 16384 blocks
unlimited inodes, largefiles supported
716603392 data blocks, 716406528 free data blocks
21869 allocation units of 32768 blocks, 32768 data blocks
# mount /oracle
#

now it is mounted. when browsing with winscp, in the folder oracle (that before was empty) appeared only a folder named lost+found but that is empty too.
now is there smth else i should do or just reinstalling oracle will finish the job?

[g3jza]

You can continue reinstalling the oracle and possibly recovering the data you need from backups.

Lost+Found is a place where orphan files (those that are not associated with any inodes) are being put, just ignore it.