ML370 - Smart Array 5300 Drive Rebuild Problem

[highrolla]

I have a Proliant ML370 with a Smart Array 5300, Firmware 3.54 that won't rebuild the array.

A drive has failed and the hot spare kicked in. When I replace failed drive with a new drive, green lights come on and flash like the drive is rebuilding, and CPQRAID reports the following:

"CPQRAID: The HP Smart Array 5300 Slot 1 ID 0 LUN 0 device is now rebuilding data on the new drive replacing the previously failed drive."

Two minutes later the LED goes off on the drive, and CPQRAID reports:

"CPQRAID: The HP Smart Array 5300 Slot 1 ID 0 LUN 0 is queued to begin rebuilding data on the new drive replacing the previously failed drive."

When you view the array status in CPQONLIN, it is reported that the rebuild is queued and it sits in this state. The drive never rebuilds and it appears that it is just hung.

Does anyone have any idea what I could try next?

Any help is greatly appreciated!

[Víctor Cespón · Accepted solution]

Usually when a RAID does not rebuild after replacing a drive, the explanation is that there is another disk that is failing.
Run an ADU report and post it here.

[highrolla]

ADU Report attached.

[Víctor Cespón]

SCSI Port 1, Drive ID 1
Serv. Time 2395778 (4.55 years)
Hrd Read 1319
Rtry Read 20
Hrd Write 1
Rtry Write 38
Fl Rd Recv 70
Product Id BF01865222 Spare: 189395-001, 18GB Wide Ultra315K rpm1 18GB 1 Wide Ultra3 15K 80P 18GB Hotplug

It has a lot of messages with "Read Retries Exhausted"

The other disks have no errors on last months.

This is why it does not rebuild:

Rebuild aborted due to read error, logical drive 00000000. Replacement drive 2 Error drive 1 Big replacement drive 0x82 Big error drive 0x81 12-04-2007 11:24:28

Rebuild aborted due to read error, logical drive 00000000. Replacement drive 2 Error drive 1 Big replacement drive 0x82 Big error drive 0x81 12-04-2007 12:09:26

[highrolla]

So, I guess we should try swapping drive in slot 1 with a good drive and see if it rebuilds... and then if it does, put new drive in slot 2 and see if it will also rebuild?

The drives in slot 3 and 0 appear to be working correctly, so swapping the drive in slot 1 should not blow away the array. Correct?

Thank you for your help!

[Víctor Cespón]

SCSI Port 1, Drive ID 1 needs replacent
The problem I see is that you also replaced SCSI Port 1, Drive ID 2. If they are part of the same RAID, the rebuild process can be compromised.

[highrolla]

Yes, Drive ID 2 is the one we replaced a few days ago that will not rebuild, evidently due to errors on Drive ID 1.

But, in this array, there is still Drive ID 0 and Drive ID 3 that are functioning properly, so I am hoping that Drive ID 1 will rebuild if we insert a good drive...

Thanks again for your help!