topic Re: Betreff: Another server, same embedded flash / SD-Card Error in ProLiant Servers (ML,DL,SL)

Another server, same embedded flash / SD-Card Error

Downshifter — Tue, 14 Feb 2023 00:49:23 GMT

Hello everyone!

Any kind of help is appreciated.

Hope dies last, and I hope to get a solution for my problem on a ProLiant ML350 Gen9 Server.

I had a similar problem with a ML350 G8 Server two weeks ago and ended up changing the mainboard of the server. G8 mainboards are cheap, G9 mainboards cost a fortune currently here in Germany.

I spent a lot of time solving the problem on different ways:

During boot the system stopped with a System Health error message:

“-338-HP RESTful API Error – Unable to communicate with iLO FW. BIOS configuration resources may not be up-to-date. Actio: Restore Manufacturing Defaults in RBSU.”

Further, on iLO logon website the erroro message “iLO Self-Test reports a problem with: Embedded Flash/SD-CARD. View details on Diagnostics page.” In iLO the health status was degraded

What I have done so far after Installing SPP P52574_001_spp-Gen9.1-Gen9SPPGen91.2022_0822.4 (the last I have access to)

Reset BIOS to factory default.
Cleared IML logs
Formatted embedded flash and reset iLO from iLO Diagnostics page.
Used dip switch 6 procedure for reset.
Created a “Force-Format.xml” file and executed it from HPQLOCFG.exe command line utility.
Did a RBSU reset to default values and lost F10 intelligent provisioning content during this procedure.
Trying to reinstall intelligent provisioning ended with RAM write error.
Completely frustrated.

After doing step 3. or 5. Everything seemed to be fine (no error messages on iLO log in website or heath state error until I do a server reboot. Then the same problem pops up again.

Is there any step I missed what could bring me closer to a solution and preventing me to buy a new server or mainboard?

Thanks in advance for help!

Andreas

Betreff: Another server, same embedded flash / SD-Card Error

Cali — Fri, 10 Feb 2023 14:32:31 GMT

Hello Andreas,

if you've tried everything you can find here Document - Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) – How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning | HPE Support and it looks like for me. Then in the end there are only two things left, either replace the mainboard or ignore it. With such an old Server I tend toward the later. I have several Servers that have been running with the error for a long time without any problems. And you also don't know the condition of a used mainboard if you don't get it originally from HPE.

Cali

Re: Betreff: Another server, same embedded flash / SD-Card Error

Tam92 — Mon, 13 Feb 2023 00:50:45 GMT

Hello,

As all the required troubleshooting steps have already been performed, to resolve this issue, the system board needs to be replaced.

Here it seems the NAND chip which is integrated on the system board is corrupted that is the reason the steps did not work.

You will not be able to access the features provided by Intelligent provisioning including the AHS logs download.

Thanks,

TAM

Betreff: Another server, same embedded flash / SD-Card Error

Downshifter — Mon, 13 Feb 2023 08:52:14 GMT

Thank you Cali,

I hate wasting electronic devieses and from my piont of view, a Gen9 server is not old at all.
It seems that I have to accept that the good old times, where expensive gear lasted a long time, are over. The world gets drown in electronic rubbish. Crazy.

Thank you for your support!

Andreas

Re: Betreff: Another server, same embedded flash / SD-Card Error

Downshifter — Mon, 13 Feb 2023 08:58:39 GMT

Thanks TAM!

Sad, but it seems that I have to face the truth. Frustrating that so many HP servers have the same issue caused by a chip worth only a coulple of dollar.

I'm dealing with HP servers since nearly 30 Years and there are still ML350 Gen6 machines out there running fine for their purposes. Where is the quality I was used to for so many years?

Andreas

Re: Betreff: Another server, same embedded flash / SD-Card Error

Tam92 — Tue, 14 Feb 2023 00:43:40 GMT

Hello,

Apologies for the inconvenience. It is not certain that all the Gen9 servers experience this issue. Intelligent provisioning was introduced from Gen8 onwards, so the requirement of NAND chip came into picture.

Thanks,

TAM

Re: Betreff: Another server, same embedded flash / SD-Card Error

MykD — Mon, 16 Sep 2024 09:22:43 GMT

Sorry to ressurect this thread, but can this error be cleared by putting an SD-Card in the SD-CARD slot to be used instead of the embedded flash? Does anyone know what is involved in doing that? Do we need a copy of the iLO config already on the SD-Card, or can we copy from the Embedded flash to the SD-Card and then switch to using it?

Re: Betreff: Another server, same embedded flash / SD-Card Error

Sunitha_Mod — Mon, 24 Jun 2024 08:06:10 GMT

Hello @MykD,

Thank you for writing to us!

You might want to consider creating a new topic by utilizing the ""New Discussion"" button, as this will not only enhance visibility compared to the old topic but also boost your chances of receiving responses from experts.

Re: Betreff: Another server, same embedded flash / SD-Card Error

Prudentcircle — Mon, 16 Sep 2024 09:33:40 GMT

There seemed to be an active discussion about this topic and the conclustion is, the NAND flash is soldered on the mainboard.

If you are skilled enough, you could try to remove it and solder it back onto the board yourself.

But, there does not seem to be an officially supported way to replace the NAND chip only.

The only option right now seems to be replacing the entire mainboard.

https://serverfault.com/questions/941972/what-is-the-nand-used-for-in-the-hpe-proliant-dl360p-gen8

Re: Betreff: Another server, same embedded flash / SD-Card Error

hunter86_bg — Sun, 19 Jan 2025 14:07:39 GMT

Seems that some users successfully replaced the NAND chip on Gen8/Gen9 systems:
https://www.reddit.com/r/homelab/comments/vikdus/has_anyone_tried_replacing_the_ilo_nand/