topic Re: SD-CARD - Embedded media initialization failed in ProLiant Servers (ML,DL,SL)

SD-CARD - Embedded media initialization failed

testmado662 — Tue, 11 Sep 2018 14:52:02 GMT

Hello,

Our Gen8 Proliant with ILO4 2.6 and last SPP, have SD-Media card problems. Error message in ILO is "Embedded media initialization failed due to media write-verify faillure. I've attached an ILO view.

We thought to have found solution with Document ID: c04996097, but we lose more and more servers. Servers with issue can be accessed by VMWare because SD CARD can't be used.

There is a lot of issues posted in 2016, 2017 with SD CARD but somebody knows what to do now with ILO4 last version and SPP last version ?

Thanks for your help

Re: SD-CARD - Embedded media initialization failed

Torsten. — Tue, 11 Sep 2018 15:09:16 GMT

If possible upgrade ILO to version 2.60.

Then go to Diagnostics, ILO health and format the NAND.

Pull the power cords, re-connect them and check in ILO Diagnostics page.

Re: SD-CARD - Embedded media initialization failed

testmado662 — Tue, 11 Sep 2018 15:38:22 GMT

Hello Torsten,

All ILO4 with the problem are currently in 2.60.

We tried a NAND fromat as ddescribe in note #c04996097 but don't work. At the end of the note, HP inform us that if all operations listed failed, we will need to change mother board. I'm surprised that no other solution exists.

Thanks

Re: SD-CARD - Embedded media initialization failed

Torsten. — Tue, 11 Sep 2018 16:16:12 GMT

Try what I told you and pull the power plugs after sending the format command.

Re: SD-CARD - Embedded media initialization failed

testmado662 — Wed, 12 Sep 2018 08:42:09 GMT

Hi Torsten,

We did it, SD-CARD was formated and power unplugged during 30 sec.

We update to 2.61, this version correct some SD-CARD issues but don't work in our case.

I attached some snapshot in order to view error message we got.

Thanks for your help

Re: SD-CARD - Embedded media initialization failed

Torsten. — Wed, 12 Sep 2018 09:01:06 GMT

The error message looks like the NAND is damaged.

If this is the case, only a system board replacement will solve the issue.

However, usually it looks like this:

and after format and pulling the power plugs you should find an entry in ILO log

(has been formatted)

Re: SD-CARD - Embedded media initialization failed

testmado662 — Wed, 12 Sep 2018 10:48:34 GMT

Hello Torsten,

Our Gen8 are not supported and I Gen8 mainboard will ave to be reserved to client with a waranty cintract.

Can we change the nan component on mainboard ?

Thanks

Re: SD-CARD - Embedded media initialization failed

Torsten. — Wed, 12 Sep 2018 10:55:06 GMT

Unlikely.

It is a soldered chip.

However, most of the times I was successful with format, even it sometimes takes several attemps.

Re: SD-CARD - Embedded media initialization failed

testmado662 — Thu, 13 Sep 2018 10:23:34 GMT

hello,

What I don't ubderstand, it's le link between SD-Card and ILO chip ?

Thanks

Re: SD-CARD - Embedded media initialization failed

Torsten. — Thu, 13 Sep 2018 10:28:33 GMT

ILO is the I/O controller for the SD-card slot.

If ILO fails to start, the SD-card slot is not available.

If you use ESX, you may consider to use the internal USB slot.

Re: SD-CARD - Embedded media initialization failed

neilvbell — Fri, 03 May 2019 15:21:20 GMT

Possibly an old issue but we keep seeing these issues across our HP estate of ESXi hosts. Losing ESXi management access, restarting agents doesn't help, can't even easily restart the host as iLO always seemed to be unavailable at the same time. I have now found a few workarounds to recover management access of these hosts without having to do a forced restart and rely on HA (painful as this will reboot VMs and require recovery).

The problem starts with the flakey iLO. When iLO fails, the SD card loses connectivity and finally ESXi management is lost as can't write to SD card. VMs all still running happily but can't migrate them off. First step is to reset the iLO via SSH. SSH to the iLO IP, login using iLO admin credentials. Type "cd /map1" (without the quotes), press Enter, then type "reset" and press Enter. This will force a reset of iLO (which often, but not always, isn't accessible anyway). Since SD card is hosted via iLO you lose access to it but you already lost that anyway. iLO should restart and this should bring your SD card back to life.

Next step is to restart the ESXi management agents, now that the SD card is alive again. I use the iLO remote console and F2, then login to ESXi as root, Troubleshooting Options, Restart Management Agents, F11 to complete. wait for the agents to restart. Go back to vcenter and your ESXi host should now be accessible again. I maintenance mode the host and reboot, just to be sure.

Hope that works and is helpful for others that might be suffering the same HP iLO pain. ;)

Re: SD-CARD - Embedded media initialization failed

neilvbell — Fri, 03 May 2019 15:47:53 GMT

If none of that works then I did find that I could do a linux reboot of the hosts via SSH. It would take a good 10 to 30 mins to take effect but finally the host would reboot and HA would take over to recover the affected VMs. Much less pleasant but at least it avoided haveing to pay for remote hands to reboot the hosts manually when i had lost access to iLO.

SSH to ESXi management IP and login as root. Type "reboot" (without quotes and press enter. Wait with fingers crossed. If and when you lose SSH access then it is finally rebooting and should come back along with iLO and SD card as if it never had a problem.

Re: SD-CARD - Embedded media initialization failed

Torsten. — Fri, 03 May 2019 15:19:25 GMT

See

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning

https://support.hpe.com/hpsc/doc/public/display?docId=a00048622en_us

Re: SD-CARD - Embedded media initialization failed

neilvbell — Fri, 03 May 2019 15:38:42 GMT

Hi Torsten,

Yes, thanks for that. I was aware of this as a potential solution for the flakey iLO/SD-card issue. I have used this once successfully and twice unsuccessfully, plus I have machines that lose SD card connectivity but don't show up in diagnostics as having the NAND error (even though on latest iLO firmware). It also doesn't help when you have a production ESXi host that has critical VMs on it and they are now orphaned (including not getting Veeam backups, since these run through vcenter management). This workaround is to allow recovery of the host so you can get the VMs back under control prior to whatever host fix you might then want to enact and since I have now used this successfully twice and have never seen it referenced anywhere I thought it might help someone.

Kind regards,

Neil

Re: SD-CARD - Embedded media initialization failed

sv000008 — Wed, 23 Apr 2025 09:21:19 GMT

Quick, easy and to the point. Took me less than 5 minutes to do everything and format the NAND. Issue resolved.

~~https://sysadminguides.org/2018/08/26/fix-ilo-hp-error-embedded-flash-sd-card-embedded-media-manager-failed-initialization/~~

Moderator [above link is no longer valid, please visit https://support.hpe.com/connect/s/ to find the latest info ]

Re: SD-CARD - Embedded media initialization failed

PDP-Fan — Wed, 29 May 2024 08:03:47 GMT

I'm sorry for coming late to this discussion but the problem is still relevant...

The iLO chip itself is not the problem but the Sandisk flash chip. This chip is in fact an SD card soldered to the mainboard. It is not clear why HP decided to use a soldered chip instead if a removable Micro-SD card.
We have successfully replaced the flash chip (with help from a professional repair shop) and that solved the problem. The re-format tool provided on the iLO webpage is not a solution. It may work for a few hours or days but when the chip is damaged, the error will come back.

A simple test for the flash chip is to try updating the Intelligent Provisioning software. This will write directly to that chip and will fail if there are any problems.

The solution for everyone without professional soldering skills is to replace the mainboard. Hopefully with one with a good chip... but you can't say how many read/writes that chip already has.