Internal SAS Enclosure Device Failure (HP DL380 G6 E5504)

[cylai]

Hi,

We have an HP DL380 G6 E5504 Server.

And recently it encounter some errors. Upon checking on the HP System Management Homepage, HP Proliant Integrated management log viewer, and OS event log. It sound like the HDD failure.

I want to confirm whether our findings is correct. If yes how to do we go about it?

From the HP Proliant integrated management log viewer, we can see

Description Class Code Severity Count Update Time Initial Time

Internal SAS Enclosure Device Failure (Bay 2, Box 1, Port 1I, Slot 0) Drive Array Subsystem 7 Critical 1 9/21/2013 11:54 PM 9/21/2013 11:54 PM

From the HP System Management Homepage

Refer to the attachement below.

From the OS System event log, we can see

10/4/2013 11:57:54 AM      Cissesrv      Error      None      24606      N/A      OL01-ORC1

"Logical drive 1 configured on array controller P410i [Embedded] returned a fatal error during a read/write request from/to the volume.

Logical block address 536457800, block count 248 and command 32 were taken from the failed logical I/O request.

Array controller P410i [Embedded] is also reporting that the last physical drive to report a fatal error condition (associated with this logical request), is located on bus 0 and ID 3. "

10/4/201311:57:36 AM       Disk      Error     None     11     N/A     OL01-ORC1

The driver detected a controller error on \Device\Harddisk0.

9/21/2013 11:54:32 PM      Storage Agents      Warning      Events      1200      N/A      OL01-ORC1

"Drive Array Logical Drive Status Change. Logical drive number 1 on the array controller in Slot 0 has a new status of 5.
(Logical Drive status values: 1=other, 2=ok, 3=failed, 4=unconfigured, 5=recovering, 6=readyForRebuild, 7=rebuilding, 8=wrongDrive, 9=badConnect, 10=overheating, 11=shutdown, 12=expanding, 13=notAvailable, 14=queuedForExpansion, 15=multipathAccessDegraded, 16=erasing)
[SNMP TRAP: 3034 in CPQIDA.MIB]"

9/21/2013 11:54:32 PM      Storage Agents      Error      Events      1216      N/A      OL01-ORC1

"Drive Array Physical Drive Status Change. The physical drive in Slot 0, Port 1I Box 1 Bay 2 with serial number ""3TB0Q6FP00009013UFFP"", has a new status of 3.
(Drive status values: 1=other, 2=ok, 3=failed, 4=predictiveFailure, 5=erasing, 6=eraseDone, 7=eraseQueued)
[SNMP TRAP: 3046 in CPQIDA.MIB]"

9/21/2013 11:54:15 PM      Cissesrv      Information      None      24598      N/A      OL01-ORC1

"Logical drive 1 of array controller P410i [Embedded] has encountered a status change from:

Status: OK
to
Status: INTERIM RECOVERY MODE"

9/21/2013 11:54:15 PM      Cissesrv      Error      None      24595      N/A      OL01-ORC1

A drive failure notification has been received for the SAS physical drive located in bay 2. This drive can be found in box 1 which is connected to port 1I of the array controller P410i [Embedded].
The failure reason received from the HP Smart Array firmware is: MARK_BAD_FAILED.

The question is, does it means physical drive error? 

If it is physical drive error, We just need to get a HDD replacement from HP and hot swap the HDD then it will rebuild the new HDD? or do we need to reset the RAID settings?

Hope anyone here can provide some advice. If you need further information. Do let me know.

Thank you in advance.

[Torsten. · Accepted solution]

This really looks like a failed disk.
You should open ACU and confirm.
Replace the disk.

[cylai]

Thanks @Torsten for your quick reply.

Attached information retrieve from HP Insifht dianostics online edition for windows.

It does seems to be HDD failure.

I can see that our RAID settings is 1+0.

For our case, we need to get a new HDD from HP and only need to hotswap the defect HDD?

Is there a need to reconfigure the RAID settings?

Thank you.

[Torsten.]

Replace the disk and the array will be synchronized automatically.

[cylai]

Thanks @Torsten.

Will try to order a replacement from HP.

[cylai]

Hi,

Following up this case.

I had purchase 4 compatible hdd for our server to be replaced.

So we did a hotswapped of the defective hdd with the new hdd.

From the ACU we can see it is rebuilding.  However it encountered rebuilding failed.

And the new HDD LED status is showing regular flashing amber color.

Check on the ACU report, it say that HDD is predictive failure status.

I then tried with another HDD. It encountered the same thing as well.

Contacted HP support. They adviced to upgrade to the latest firmware and check the cabling.

We did that. And tried rebuilding with the new HDD. The amber LED status still showing.

Anyone encountered such issue and how to solved this?

[cylai]

Hi,

Attached along is the ADU report.

I found something suspicious and i would like to have some confirmation from fellow contributor here.

From ADU report at the "Monitor and Performance Statistics (Since Factory)" section. We find 

Read Errors Hard 0x00000005
Read Errors Retry Recovered 0x00000000
Write Errors Hard 0x00000000
Write Errors Retry Recovered 0x00000000
Other Time Outs 0x0000

where else at the "Monitor and Performance Statistics (Since Reset)" section. We find 

Read Errors Hard 0x00000000
Read Errors Retry Recovered 0x00000000
Write Errors Hard 0x00000000
Write Errors Retry Recovered 0x00000000
Other Time Outs 0x0000

According to this article 

https://support.hpe.com/hpesc/public/docDisplay?docId=sf000055442en_us&docLocale=en_US

The article mention that the above hexadecimal numbers should always be in 0's. Any thing other that that indicates a potential failure.

So my question is. What is the difference between "Since Factory" and "Since Reset'?

If "Since Factory" means brand new unit, does it mean that the HDD is already defective since brand new?

Thank you. Hope to get some reply.

[Note: broken link updated/removed by Mod]

[RattanKanna]

Hey Guys.. 

I am facing a similar issue on a HP ProLiant SL4540 Gen8 Tray 3x Node Server

"Internal Storage Enclosure Device Failure (Bay 11, Box 1, Port 2I, Slot 1)"
"Internal Storage Enclosure Device Failure (Bay 12, Box 1, Port 2I, Slot 1)"
"Internal Storage Enclosure Device Failure (Bay 13, Box 1, Port 2I, Slot 1)"
"Internal Storage Enclosure Device Failure (Bay 14, Box 1, Port 2I, Slot 1)"
"Internal Storage Enclosure Device Failure (Bay 15, Box 1, Port 2I, Slot 1)"

The hard drives are working fine on a known good server 

Smart Array P420i mezzanine controller, FBWC, Capacitor, are replaced. once the parts are replaced. i am facing simiar issue

could some one please advise 

[cylai]

Hi RattanKanna,

I'm not sure about your case FBWC, Capacitor.

For my case, the replacement HDD is defective before i installed it to the server.

You can look at the post on ‎12-15-2013 07:09 PM

Replaced the defective drive then my case is solved.

Hope you can get yours fixed soon.

Do contact HP support. They are some what quite supportive.

Prepared ADU report. They will request for it. And try to read and understand the ADU report yourself if can.

You may find the root cause yourself like for my case.

Peace.