SD-CARD - Embedded media initialization failed

[neilvbell]

Possibly an old issue but we keep seeing these issues across our HP estate of ESXi hosts. Losing ESXi management access, restarting agents doesn't help, can't even easily restart the host as iLO always seemed to be unavailable at the same time. I have now found a few workarounds to recover management access of these hosts without having to do a forced restart and rely on HA (painful as this will reboot VMs and require recovery).

The problem starts with the flakey iLO. When iLO fails, the SD card loses connectivity and finally ESXi management is lost as can't write to SD card. VMs all still running happily but can't migrate them off. First step is to reset the iLO via SSH. SSH to the iLO IP, login using iLO admin credentials. Type "cd /map1" (without the quotes), press Enter, then type "reset" and press Enter. This will force a reset of iLO (which often, but not always, isn't accessible anyway). Since SD card is hosted via iLO you lose access to it but you already lost that anyway. iLO should restart and this should bring your SD card back to life.

Next step is to restart the ESXi management agents, now that the SD card is alive again. I use the iLO remote console and F2, then login to ESXi as root, Troubleshooting Options, Restart Management Agents, F11 to complete. wait for the agents to restart. Go back to vcenter and your ESXi host should now be accessible again. I maintenance mode the host and reboot, just to be sure.

Hope that works and is helpful for others that might be suffering the same HP iLO pain. ;)

[neilvbell]

If none of that works then I did find that I could do a linux reboot of the hosts via SSH. It would take a good 10 to 30 mins to take effect but finally the host would reboot and HA would take over to recover the affected VMs. Much less pleasant but at least it avoided haveing to pay for remote hands to reboot the hosts manually when i had lost access to iLO.

SSH to ESXi management IP and login as root. Type "reboot" (without quotes and press enter. Wait with fingers crossed. If and when you lose SSH access then it is finally rebooting and should come back along with iLO and SD card as if it never had a problem.

[Torsten.]

See

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning

https://support.hpe.com/hpsc/doc/public/display?docId=a00048622en_us

[neilvbell]

Hi Torsten,

Yes, thanks for that. I was aware of this as a potential solution for the flakey iLO/SD-card issue. I have used this once successfully and twice unsuccessfully, plus I have machines that lose SD card connectivity but don't show up in diagnostics as having the NAND error (even though on latest iLO firmware). It also doesn't help when you have a production ESXi host that has critical VMs on it and they are now orphaned (including not getting Veeam backups, since these run through vcenter management). This workaround is to allow recovery of the host so you can get the VMs back under control prior to whatever host fix you might then want to enact and since I have now used this successfully twice and have never seen it referenced anywhere I thought it might help someone.

Kind regards,

Neil

[sv000008]

Quick, easy and to the point. Took me less than 5 minutes to do everything and format the NAND. Issue resolved.

https://sysadminguides.org/2018/08/26/fix-ilo-hp-error-embedded-flash-sd-card-embedded-media-manager-failed-initialization/