HP DL380 G6 RAID 1+0 - 2 drives, both predicted failure - Will not rebuild

[omega1]

Hi there,

I have inherited a HP DL360 G6 that is displaying 'Predicted Failure' on both of the two drives in the RAID array.

I attended the server location today with a new drive and HOT swap replaced it nad was hoping for the array to be rebuilt, but it stopped at 4% of the rebuild process and then went to 'Ready to rebuild' and marked the new drive as 'predicted failure'.

I wasn't watching the process the first time round, but took out the new drive and put it back in again and it did the same thing. Failing again at 4%.

I suspect that I am now going to have to perform a re-install due to drive errors, but as the server is still online, I'd like to try and avoid this.

Is there a process I can follow to recover the system? Obviously I'll be backing up the data, but I was wondering if I could add any additional drives in the other slots to have it rebuild there? Or is it unlikely that I can recover the system? I suspect if the server reboots it will not restart.

I'm happy to provide more detailed errors on request.

Device Diagnosis for Logical Volume 0, Controller Slot 0 Bus 0
Physical Hard Drive 1, Serial Number: Y3H0A0QCFTM91346, Controller Serial Number: 50123456789ABCDE

Failed
Error: 640001: Controller has reported a SMART error on this drive
Error: 640006: The Read and/or Write HARD error rate is above threshold

Accumulated power-on time for this hard drive: 25.7 months
This drive has experienced/recorded error conditions reported by diagnosis and requires replacement.

Perform the following steps before replacing a drive:

Have a known good data backup.
Replace the failed drive with one that is the same as, or functionally equivalent to, the original drive.
Do not replace the drive if there is another drive that is failed, offline, or in the process of being rebuilt.
On non-Hot Plug drives make sure the replacement drive ID matches the original drive ID.
For further replacement procedures or help with identifying drive status, see the HP Smart Array Controller User Guide for your controller. The User Guide is available from hp.com.

To identify the failed drive, click on the 'Identify Drive' button below to flash the LED on this drive.

Clicking on Identify Drive button will blink the amber LED (if SCSI drive) or turn ON the blue LED (if SAS or SATA drive) for a few seconds.

Identify Drive

Physical Hard Drive 2, Serial Number: S0K2JQHS0000M444T9HF, Controller Serial Number: 50123456789ABCDE

Failed
Error: 640001: Controller has reported a SMART error on this drive
Error: 640006: The Read and/or Write HARD error rate is above threshold

Accumulated power-on time for this hard drive: 18.5 months
This drive has experienced/recorded error conditions reported by diagnosis and requires replacement.

Perform the following steps before replacing a drive:

Have a known good data backup.
Replace the failed drive with one that is the same as, or functionally equivalent to, the original drive.
Do not replace the drive if there is another drive that is failed, offline, or in the process of being rebuilt.
On non-Hot Plug drives make sure the replacement drive ID matches the original drive ID.
For further replacement procedures or help with identifying drive status, see the HP Smart Array Controller User Guide for your controller. The User Guide is available from hp.com.