MCTP error that leads to NVMe drives failing

[DrThrax]

Hello
we have 4 DL380 gen 10 servers that use NVMe drives and randomly show an error on IML that says:
"Innovation Engine not responding to MCTP Requests, please AC Cycle the server to recover IE." and then proceed to fail one or 2 of the NVMe drives on the server

and so far I have done the following steps in order to fix the problem but no luck:

1-applied SPP and AC cycle

2-updated IE and ROM firmware to the latest version and ILO 5 reset + update ( IE: 0.2.3.0 and BIOS: 2.72 and ILO: 2.65 )

3-moved the failed drives on other servers but the error still presists and another drive fails randomly

[support_s]

System recommended content:

1. Advisory: (Revision) HPE NVMe SSDs - Excessive MCTP Warning Messages in iLO Event Log for HPE PCIe NVMe Server Solid State Drives (SSDs)

2. Advisory: (Revision) HPE Integrated Lights-Out 5 (iLO 5) - OneView Profile May Fail to Apply and iLO User Interface May Become Unresponsive on HPE Gen10-series Server with an Installed NVMe Drive and/or Attached SPP and Managed BIOS

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[DrThrax]

hello and thank you for the response

the firmware of the NVMe drives are GPK5 and MPK76H5Q which are not in the page link that you provided.

the second link is not related to the problem but I have tested it and it didnt work.

Regards

Sohrab Khani

[Bunsol]

DrThrax,

I have sent a private message to you for this issue. Kindly log a support case for this issue.

[EB3]

@Bunsol 

Hi,

I have the exact same issue on a HPE DL385 Gen10 plus v2.

NVME SSD's: MZXLR800HBHQ-000H3
Firwmare version: MPK7625Q

All 4 drives are NVMe and have a failed status.

[GabbeB]

I have the same issue, did someone solve this?

[EB3]

I just opened a case with HPE, waiting for techsupport for a reply. Will keep you posted here once I receive the solution.

[GabbeB]

new ILO firmware was released, i have applied this to a couple of servers and erros is gone so far

https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_6ae65df0866f4f8eaf067d0a7e&tab=revisionHistory

[EB3]

Thanks, just did the update.