SD-CARD - Embedded media initialization failed

[testmado662]

Hello,

Our Gen8 Proliant with ILO4 2.6 and last SPP, have SD-Media card problems. Error message  in ILO is "Embedded media initialization failed due to media write-verify faillure. I've attached an ILO view.

We thought to have found solution with Document ID: c04996097, but we lose more and more servers. Servers with issue can be accessed by VMWare because SD CARD can't be used.

There is a lot of issues posted in 2016, 2017 with SD CARD but somebody knows what to do now with ILO4 last version and SPP last version ?

Thanks for your help

[Torsten.]

If possible upgrade ILO to version 2.60.

Then go to Diagnostics, ILO health and format the NAND.

Pull the power cords, re-connect them and check in ILO Diagnostics page.

[testmado662]

Hello Torsten,

All ILO4 with the problem are currently in 2.60.

We tried a NAND fromat as ddescribe in note #c04996097  but don't work. At the end of the note, HP inform us that if all operations listed failed, we will need to change mother board. I'm surprised that no other solution exists.

Thanks

[Torsten.]

Try what I told you and pull the power plugs after sending the format command.

[testmado662]

Hi Torsten,

We did it, SD-CARD was formated and power unplugged during 30 sec.

We update to 2.61, this version correct some SD-CARD issues but don't work in our case.

I attached some snapshot in order to view error message we got.

Thanks for your help

[Torsten.]

The error message looks like the NAND is damaged.

If this is the case, only a system board replacement will solve the issue.

However, usually it looks like this:

and after format and pulling the power plugs you should find an entry in ILO log

(has been formatted)

[testmado662]

Hello Torsten,

Our Gen8 are not supported and I Gen8 mainboard will ave to be reserved to client with a waranty cintract.

Can we change the nan component on mainboard ?

Thanks

[Torsten.]

Unlikely.

It is a soldered chip.

However, most of the times I was successful with format, even it sometimes takes several attemps.

[testmado662]

hello,

What I don't ubderstand, it's le link between SD-Card and ILO chip ?

Thanks

[Torsten.]

ILO is the I/O controller for the SD-card slot.

If ILO fails to start, the SD-card slot is not available.

If you use ESX, you may consider to use the internal USB slot.

[neilvbell]

Possibly an old issue but we keep seeing these issues across our HP estate of ESXi hosts. Losing ESXi management access, restarting agents doesn't help, can't even easily restart the host as iLO always seemed to be unavailable at the same time. I have now found a few workarounds to recover management access of these hosts without having to do a forced restart and rely on HA (painful as this will reboot VMs and require recovery).

The problem starts with the flakey iLO. When iLO fails, the SD card loses connectivity and finally ESXi management is lost as can't write to SD card. VMs all still running happily but can't migrate them off. First step is to reset the iLO via SSH. SSH to the iLO IP, login using iLO admin credentials. Type "cd /map1" (without the quotes), press Enter, then type "reset" and press Enter. This will force a reset of iLO (which often, but not always, isn't accessible anyway). Since SD card is hosted via iLO you lose access to it but you already lost that anyway. iLO should restart and this should bring your SD card back to life.

Next step is to restart the ESXi management agents, now that the SD card is alive again. I use the iLO remote console and F2, then login to ESXi as root, Troubleshooting Options, Restart Management Agents, F11 to complete. wait for the agents to restart. Go back to vcenter and your ESXi host should now be accessible again. I maintenance mode the host and reboot, just to be sure.

Hope that works and is helpful for others that might be suffering the same HP iLO pain. ;)

[neilvbell]

If none of that works then I did find that I could do a linux reboot of the hosts via SSH. It would take a good 10 to 30 mins to take effect but finally the host would reboot and HA would take over to recover the affected VMs. Much less pleasant but at least it avoided haveing to pay for remote hands to reboot the hosts manually when i had lost access to iLO.

SSH to ESXi management IP and login as root. Type "reboot" (without quotes and press enter. Wait with fingers crossed. If and when you lose SSH access then it is finally rebooting and should come back along with iLO and SD card as if it never had a problem.

[Torsten.]

See

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning

https://support.hpe.com/hpsc/doc/public/display?docId=a00048622en_us

[neilvbell]

Hi Torsten,

Yes, thanks for that. I was aware of this as a potential solution for the flakey iLO/SD-card issue. I have used this once successfully and twice unsuccessfully, plus I have machines that lose SD card connectivity but don't show up in diagnostics as having the NAND error (even though on latest iLO firmware). It also doesn't help when you have a production ESXi host that has critical VMs on it and they are now orphaned (including not getting Veeam backups, since these run through vcenter management). This workaround is to allow recovery of the host so you can get the VMs back under control prior to whatever host fix you might then want to enact and since I have now used this successfully twice and have never seen it referenced anywhere I thought it might help someone.

Kind regards,

Neil

[sv000008]

Quick, easy and to the point. Took me less than 5 minutes to do everything and format the NAND. Issue resolved.

https://sysadminguides.org/2018/08/26/fix-ilo-hp-error-embedded-flash-sd-card-embedded-media-manager-failed-initialization/ 

Moderator [above link is no longer valid, please visit https://support.hpe.com/connect/s/  to find the latest info ]

[PDP-Fan]

I'm sorry for coming late to this discussion but the problem is still relevant...

The iLO chip itself is not the problem but the Sandisk flash chip. This chip is in fact an SD card soldered to the mainboard. It is not clear why HP decided to use a soldered chip instead if a removable Micro-SD card.
We have successfully replaced the flash chip (with help from a professional repair shop) and that solved the problem. The re-format tool provided on the iLO webpage is not a solution. It may work for a few hours or days but when the chip is damaged, the error will come back.

A simple test for the flash chip is to try updating the Intelligent Provisioning software. This will write directly to that chip and will fail if there are any problems.

The solution for everyone without professional soldering skills is to replace the mainboard. Hopefully with one with a good chip... but you can't say how many read/writes that chip already has.