Betreff: Another server, same embedded flash / SD-Card Error

[Downshifter]

Hello everyone!

Any kind of help is appreciated.

Hope dies last, and I hope to get a solution for my problem on a ProLiant ML350 Gen9 Server.

I had a similar problem with a ML350 G8 Server two weeks ago and ended up changing the mainboard of the server. G8 mainboards are cheap, G9 mainboards cost a fortune currently here in Germany.

I spent a lot of time solving the problem on different ways:

During boot the system stopped with a System Health error message:

 “-338-HP RESTful API Error – Unable to communicate with iLO FW. BIOS configuration resources may not be up-to-date. Actio: Restore Manufacturing Defaults in RBSU.”

Further, on iLO logon website the erroro message “iLO Self-Test reports a problem with: Embedded Flash/SD-CARD. View details on Diagnostics page.” In iLO the health status was degraded

What I have done so far after Installing SPP P52574_001_spp-Gen9.1-Gen9SPPGen91.2022_0822.4 (the last I have access to)

1. Reset BIOS to factory default.
2. Cleared IML logs
3. Formatted embedded flash and reset iLO from iLO Diagnostics page.
4. Used dip switch 6 procedure for reset.
5. Created a “Force-Format.xml” file and executed it from HPQLOCFG.exe command line utility.
6. Did a RBSU reset to default values and lost F10 intelligent provisioning content during this procedure.
7. Trying to reinstall intelligent provisioning ended with RAM write error.
8. Completely frustrated.

After doing step 3. or 5. Everything seemed to be fine (no error messages on iLO log in website or heath state error until I do a server reboot. Then the same problem pops up again.

Is there any step I missed what could bring me closer to a solution and preventing me to buy a new server or mainboard?

Thanks in advance for help!

Andreas

[Cali]

Hello Andreas,

if you've tried everything you can find here Document - Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) – How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning | HPE Support and it looks like for me. Then in the end there are only two things left, either replace the mainboard or ignore it. With such an old Server I tend toward the later. I have several Servers that have been running with the error for a long time without any problems. And you also don't know the condition of a used mainboard if you don't get it originally from HPE.

Cali

[Tam92 · Accepted solution]

Hello,

As all the required troubleshooting steps have already been performed, to resolve this issue, the system board needs to be replaced.

Here it seems the NAND chip which is integrated on the system board is corrupted that is the reason the steps did not work.

You will not be able to access the features provided by Intelligent provisioning including the AHS logs download.

Thanks,

TAM

[Downshifter]

Thank you Cali,

I hate wasting electronic devieses and from my piont of view, a Gen9 server is not old at all.
It seems that I have to accept that the good old times, where expensive gear lasted a long time, are over. The world gets drown in electronic rubbish. Crazy.

Thank you for your support!

Andreas

[Downshifter]

Thanks TAM!

Sad, but it seems that I have to face the truth. Frustrating that so many HP servers have the same issue caused by a chip worth only a coulple of dollar.

I'm dealing with HP servers since nearly 30 Years and there are still ML350 Gen6 machines out there running fine for their purposes. Where is the quality I was used to for so many years?

Andreas

[Tam92]

Hello,

Apologies for the inconvenience. It is not certain that all the Gen9 servers experience this issue.  Intelligent provisioning was introduced from Gen8 onwards, so the requirement of NAND chip came into picture. 

Thanks,

TAM

[MykD]

Sorry to ressurect this thread, but can this error be cleared by putting an SD-Card in the SD-CARD slot to be used instead of the embedded flash? Does anyone know what is involved in doing that? Do we need a copy of the iLO config already on the SD-Card, or can we copy from the Embedded flash to the SD-Card and then switch to using it?

[Sunitha_Mod]

Hello @MykD,

Thank you for writing to us!

You might want to consider creating a new topic by utilizing the ""New Discussion"" button, as this will not only enhance visibility compared to the old topic but also boost your chances of receiving responses from experts.

[Prudentcircle]

There seemed to be an active discussion about this topic and the conclustion is,  the NAND flash is soldered on the mainboard. 

If you are skilled enough, you could try to remove it and solder it back onto the board yourself. 

But, there does not seem to be an officially supported way to replace the NAND chip only. 

The only option right now seems to be replacing the entire mainboard. 

https://serverfault.com/questions/941972/what-is-the-nand-used-for-in-the-hpe-proliant-dl360p-gen8

[hunter86_bg]

Seems that some users successfully replaced the NAND chip on Gen8/Gen9 systems:
https://www.reddit.com/r/homelab/comments/vikdus/has_anyone_tried_replacing_the_ilo_nand/