HPE Smart Array P824i-p MR Gen10 half of the drives get disconnected

[ak-rb01]

We have a ProLiant DL580 Gen10 server with a HPE Smart Array P824i-p MR Gen10. There are two SSDs and 16 hard drives connected to the storage controller.  After several hours or sometimes days of operation, half of the drives get disconnected from the controller and it is not possible to reconnect them. The controller does not see them anymore until we power off the server and restart. Then everything is ok until the whole problem starts again.

The firmware of the controller is version 24.23.0-0043. The drives are Model EG002400JWJNN with 2.4 TB capacity und Firmware HPD3.

/opt/MegaRAID/MegaCli/MegaCli64 -PDGetNum -aALL
Number of Physical Drives on Adapter 0: 10
Exit Code: 0x0a
/opt/MegaRAID/MegaCli/MegaCli64 -PDGetMissing -aALL
    Adapter 0 - No Missing Drive is Found.
Exit Code: 0x00

Messages in event log:
seqNum: 0x0000336d
Time: Sat May 21 03:02:59 2022

Code: 0x00000152
Class: 0
Locale: 0x20
Event Description: Controller requests a host bus rescan
Event Data:
===========
None


seqNum: 0x0000336e
Time: Sat May 21 03:03:00 2022

Code: 0x0000010c
Class: 1
Locale: 0x02
Event Description: Drive 03(e250/Port 5I Box 1 Bay 1) Path 50000398f829fa9a  reset (Type 03)
Event Data:
===========
Device ID: 3
Enclosure Index: 250
Slot Number: 1
Error: 3

seqNum: 0x0000336f
Time: Sat May 21 03:03:00 2022

Code: 0x00000070
Class: 1
Locale: 0x02
Event Description: Removed: Drive 03(e250/Port 5I Box 1 Bay 1)
Event Data:
===========
Device ID: 3
Enclosure Index: 250
Slot Number: 1


seqNum: 0x00003370
Time: Sat May 21 03:03:00 2022

Code: 0x000000f8
Class: 0
Locale: 0x02
Event Description: Removed: Drive 03(e250/Port 5I Box 1 Bay 1) Info: enclPd=fa, scsiType=0, portMap=00, sasAddr=50000398f829fa9a,0000000000000000
Event Data:
===========
Device ID: 3
Enclosure Device ID: 250
Enclosure Index: 2
Slot Number: 1
SAS Address 1: 50000398f829fa9a
SAS Address 2: 0

seqNum: 0x00003371
Time: Sat May 21 03:03:00 2022

Code: 0x000000a6
Class: 2
Locale: 0x04
Event Description: Enclosure Drive 250(Port 5I/Box 1) communication lost
Event Data:
===========
Device ID: 250
Enclosure Index: 1
Slot Number: 5


seqNum: 0x00003372
Time: Sat May 21 03:03:00 2022

Code: 0x00000072
Class: 0
Locale: 0x02
Event Description: State change on Drive 03(e250/Port 5I Box 1 Bay 1) from JBOD(40) to UNCONFIGURED_BAD(1)
Event Data:
===========
Device ID: 3
Enclosure Index: 250
Slot Number: 1
Previous state: 64
New state: 1

What could be the problem?

Thank you for your help,

Andreas

[Suman_1978]

Hi,

I would recommend to run the Diagnostics on the MegaRAID controller and share it here or you may also log a support case after obtaining the logs.

Here is the video on the same.

Thank You!
I work with HPE but opinions expressed here are mine.
Recent Support Video Releases

[ak-rb01]

@Suman_1978 

From the diagnostics it looks like we need to change the cable from the controller to one of the enclosures. This is what we will try next.

Thank you,

Andreas

[ksram]

Hi Andreas,

Let us know the status once Cables are changed.

Thank you
RamKS

[Suman_1978 · Accepted solution]

Hello @ak-rb01 

Let me know if you were able to resolve the issue.

If you have no further query and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Thankyou
Suman

[ak-rb01]

We have not received the cable yet. It will take about 2 more weeks or so.