Raid replacement drive wont rebuild

[ChrisCI]

Hi, I am running a HP DL740 server with smart array 6400 raid in raid5. My problem is one of the 4 drives in the array has failed and the replacement drive will not build. After about eight hours the recovery stops with a read error, full message below:

ciss1: *** Hot-plug drive removed: SCSI port 1 ID 2
ciss1: *** Physical drive failure: SCSI port 1 ID 2
ciss1: *** State change, logical drive 0
ciss1: logical drive 0 (da0) changed status ready for recovery->interim recovery, spare status 0x0
ciss1: *** Hot-plug drive inserted: SCSI port 1 ID 2
ciss1: *** State change, logical drive 0
ciss1: logical drive 0 (da0) changed status interim recovery->ready for recovery, spare status 0x0
ciss1: *** State change, logical drive 0
ciss1: logical drive 0 (da0) changed status ready for recovery->recovering, spare status 0x0
ciss1: *** SCSI bus speed downshifted, SCSI port 1
ciss1: *** State change, logical drive 0
ciss1: logical drive 0 (da0) changed status recovering->ready for recovery, spare status 0x0
ciss1: *** Rebuild aborted due to read error, logical drive 0
ciss1: rebuild of logical drive 0 (da0) failed due to read error

any ideas how I can get this drive to rebuild?

thanks,
Chris

[TTr]

> ciss1: rebuild of logical drive 0 (da0) failed due to read error

This is the cause. There is a read error in logical drive0 somewhere in the 3 drives. This is a block read error not a disk failure. I hope I am wrong but it looks like you might not be able to rebuild the new drive.

Take an OS level backup of everything that is on the remaining 3 disks right away. The read error might show up in some file somewhere, or it might not show up at all if the disk block where the error is has no data on it.

[ChrisCI]

Hi TTr,

Would some kind of check disk help in this case? I don't think the HP smart array utility has any disk utilities that would do this though.

Thanks for the reply.

Chris

[TTr · Accepted solution]

I don't know of any specific utility but anything that reads the entire disk image would do it. If you know Linux/UNIX something like the "dd" command. Chkdsk only checks the file and directory structures, it does not read the entire file or disk space. And the error may be in a block that does not contain any files and it will not be read.
The array reads the entire drive space, at block level, to rebuild the 4th disk. And if you have not used the entire lun0, the error may even be in a block that is not on the OS partition(s) and may not show up in the OS based file and disk reads.

[cnb]

Hi,

Can you run an ADU report and post?

Yes you have a read error somewhere in the array. ADU may be able to tell where the issue is and lend a clue on recovery method.

However, many rebuild & error recovery issues *may be* resolved with updated controller & disk firmware, drivers, and ACU/ADU application software. Older 6400 firmware is problematic and documented throughout this site.

Start here:
http://h20000.www2.hp.com/bizsupport/TechSupport/ProductList.jsp?lang=en&cc=us&taskId=135&prodTypeId=329290&prodSeriesId=367226



HTH,

[ChrisCI]

Hi cnb,

Thanks for your reponse, I will have to schedule an outage to be able to run the ADU report. The box in question is a fairly busy mail server running freebsd and will need to be taken offline. Any idea how long an ADU report takes to run over an 800GB partition?

Cheers,
Chris

[ChrisCI]

Hi CNB,

sorry for the long delay, attached is the ADU report, not sure how to read this.

Thanks,
Chris

[cnb]

Hi Chris,

Sorry...I just now saw this reply and the issue is most likely resolved?

ADU says it is rebuilding:

ROM Firmware Revision 2.80
ADU Version 8.10.3.0
Diagnostic Module Version 4.9-85
Time Generated Monday February 01, 2010 11:54:25PM

Device Summary:
Smart Array 5i in slot 0
Smart Array 6400 in slot 3
Smart Array 6400 EM in slot 3b

Consolidated Error Report:
Controller: Smart Array 5i in slot 0
Message: The cache for this controller is not configured
Controller: Smart Array 6400 in slot 3
Device: Physical Drive 1:2
Message: The data on the physical drive is being rebuilt.
Controller: Smart Array 6400 in slot 3
Device: Logical Drive 1
Message: The array controller is rebuilding this logical drive
Controller: Smart Array 6400 EM in slot 3b
Message: The cache for this controller is not configured


However, I do see some issues with the drives being of mixed Vendor/Geometry which IMHO never a good idea. The older Smart Array controllers were notorious for having rebuilding issues even when using supported HP disks and Firmware. This array in question has SEAGATE and Hitachi drives mixed in.

You do need to upgrade your Smart Array Firmware to fix some rebuild issues:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1132643&swItem=MTX-d953e02acf304fa4bc2c893b8b&prodNameId=378442&swEnvOID=181&swLang=8&taskId=135&mode=4&idx=2

Hope this helps and sorry for not seeing this sooner.

Rgds,