RAID6 doesn't rebuild

[a13x]

Hello!

DL580 G4
RAID6
8 Physical Disks 146GB

2 weeks ago I replaced one drive in the Bay4 that was blinking amber. 4 days ago I replaced the second drive in the Bay1 that was blinking amber too.

Then I restarted the server (OS is RHEL5.5) and hit F1 to start data recovery.
After this I booted from SmartStart.
At first ACU was showing that disks at the Bay 1 and 4 were recovering, but then ACU wrote:
(Ready for Recovery) The logical drive is queued for rebuilding.

If I restart the server, the situation simply repeats.

How can I force the recovery of the array?
I changed the controller settings to "High Rebuild" priority but that didn't seem to do much.

I've attached a copy of the ADU report.

Any help would be greatly appreciated.

I'm looking forward to your prompt reply.
Alex.

С Ñ Ð²Ð°Ð¶ÐµÐ½Ð¸ÐµÐ¼,
Ð Ð»ÐµÐºÑ Ð°Ð½Ð´Ñ Ð Ð¾Ð¿Ð¾Ð²,
IPG

[a13x]

Hello again!
Is there anyone who can help me?

Thanks in advance!
Alex

[Johan Guldmyr]

Hey!

After you press F1 - does it just reboot or what happens?

[a13x]

Hey!
Thank you for your reply!

When I press F1 it says that automatic data recovery enabled and the server starts to boot the OS or SmartStart (if I insert the SmartStart CD).
Then I look at the ACU. At first ACU shows that disks at the Bay 1 and 4 are recovering, but then ACU writes:
(Ready for Recovery) The logical drive is queued for rebuilding.

[Johan Guldmyr · Accepted solution]

Hey, from the ADU report it says that three drives have predictive failure.

1:6
1:7
1:8

And as far as I can see you only have one logical drive.

Can you putACU inside the OS? Maybe it is doing something different once it boots to the OS?

[a13x]

OK! I'll try and report the results!

[cnb]

Hi,

First, you can't force it because it still has an error.

Second, your P400 subsystem is missing several mandatory *critical* firmware updates and some that address issues with rebuilds and data loss, but this is not the main reason for the failure:


Smart Array P400 in slot 4 : Identify Controller

Configured Logical Drives 1 (0x01)
Configuration Signature 0xa35bd66d
RAM Firmware Revision 2.08
ROM Firmware Revision 2.08

The latest Firmware Update DVD for your Server and components:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1137825&swItem=MTX-3380ae888f86436aac54c149eb&prodNameId=3288122&swEnvOID=4006&swLang=8&taskId=135&mode=3

The rebuild is failing due to a Critical Read Error:

0 *Rebuild Aborted From Read Error Critical*

In addition to the predictive fault codes:
1I:1:6 Predictive Failure
1I:1:7 Predictive Failure
1I:1:8 Predictive Failure

Other drives reported faults and one looks like it is still reporting a fault or this is latched from tha last read failure:

1I:1:6 Last Failure Reason Hardware Error (0x0d)
2I:1:1 Last Failure Reason Hot Removed (0x14)
2I:1:4 Last Failure Reason Hot Removed (0x14)

The drives are also missing important firmware updates that affect hot removal and premature failure reporting of drives.

You're also mixing drive types which isn't the best practice and may cause rebuild failures. Try to keep the same drive type *and* firmware across the volume:

HP DG146ABAB4
HP DG146ABAB4
HP DG146ABAB4
HP DG146ABAB4
HP EG0146FAWHU *
HP DG146ABAB4
HP DG146ABAB4
HP DG0146FARVU *

1) Fix the drive at 1I:1:6
2) Update system Bios and backplane Firmware
3) Update Controller Firmware
4) Update Drive Firmware
Try the rebuild, but it probably won't work due to many fimrware issues noted.
5) Update any Linux drivers (check ADU version too!)
6) Recreate the RAID6 volume and restore from backup


Rgds,


-cnb

[cnb]

sorry, this got cut out from my last post...

"Check the Predictive or Read Faiilure Errors in ADR after updating everything. If you still get errors replace the disks with like drives, update firmware and then try the rebuild again.

If it still fails with known good replacement disks and firmware is all up to date, recreate the raid and restore."

Rgds,

[a13x]

Hello CNB and Johan Guldmyr!

The firmware is updated and the array seems to be rebuilt, but I'm not sure that it is ok.

HPACUCLI shows some new errors now.
I attach new ADU report to this message.

I'm very hopeful to save this array and it's data. Is it possible?

Thank you for your great help!
Looking forward to your reply.
A13x.

[cnb]

Your Controller Firmware is still low. 7.22 is the latest!

Your drives are still mixed, so not sure what you want analyzed. They are still reporting Predictive failures and mixed firmware.

The data should be available. Back it up, fix the drives and update the firmware.

Rgds,