reboot caused two drives to go predictive failure, replace one and stuck on "ready to rebui

[rdirkmatt]

overnight reboot caused two HDs to show status of "predictive failure" in slot's 1 and 2

I purchased two new HDs

I select the HD with more errors (slot 2 - failed recovery reads=10, other timeouts=10, hard read errors=8 and pulled that HD out and replaced it with one of the new HDs

Unfortunately for me I now have logical HD status = "Ready for Rebuild" and Parity Initialization Status = "Intialization Failed"

So from what I am reading I am screwed.

The controller firmware is 5.70 and the new drive is NOT showing any errors.  So I am guessing that the two remaining drives do not have clean parity information to use for rebuilding.

I have a good backup (storagecraft) from just prior to the problem, and also continue to get good backups (another full scheduled for tonight and incrementals for the rest of the week).

Any good options to try and get the array to rebuild, or just suck it up and flatten the box, update all firmware (bios, controller and HDs) and go for the restore?

ACU diag report attached (after drive 2 replaced)

Looking for other professional opinions on which way to proceed - thanks Dirk

[sfora]

Do you have another spare disk? I would first try with another disk. Maybe the new disk is just broke.

[waaronb]

As far as I know, it *should* have started to rebuild onto the one new drive you installed.  If the array itself became unrecoverable due to multiple drive failures, the entire array would be offline, but since you can still use it, that doesn't seem to be the case.

I'm not sure why it didn't start to rebuild on it's own unless the replacement drive wasn't large enough to replace the old one?  Usually when you get a replacement HP drive, even if it's a different model with slightly different specs, it should still work as a replacement.  I've even had rare cases where HP replaced a drive under warranty with the next larger size (300 instead of 146, for example) if they didn't have any of the smaller ones in stock.

I didn't look at your array info but you might check on there just to see if the new drive is at least as big as the other ones in the array, in case that's a problem.

You could also try rebooting the server to see if it'll start to rebuild.

Or maybe it's just having a problem with multiple drives in pre-failure mode and won't rebuild on it's own when only one is replaced.  I'm trying to remember if I've ever found myself in that situation to know if it worked normally or not, but I don't think I've had that happen before.

Keep making those backups as often as you can, and good luck.