How to recover failed iLO4 (DL380p Gen8) ?

[PC23]

Hi,

Have a situation where SUM (running from SPP P03093_001_spp-Gen8.1-SPPGen81.4) seems to be breaking iLO4, possibly bad flash, I'm not sure. This is happening on out of support DL380p Gen8 servers.

Once the update in the swpackages repository has run, it fails, then after that communication with iLO4 is not possible. I ran this on one server, worked fine and completed and everything updated and working ok. Ran it on a second server, the problem occured. I tried to manually recover the iLO4 on this second server (details later) but this did not work. I thought perhaps it was just this second server that had some problem with iLO so I started the update again on a 3rd server, the same thing happened as the second one, so now I have 2 DL380p Gen8 with dead iLO4.

Once the problem occurs the symptoms are:

- POST errors about unable to communicate with iLO4, long POST time

- Typical fan issues due to iLO problems

- When attempting to access iLO with Online configurator in Windows, error states something similar to 'unable to access security', or something

- When attempting to run SUM again (or invidical cp... iLO firmware update .exe) SUM says 'CHIF needed' and cannot do the flash. The individal .exe firmware update says it cannot find iLO in the system. I have reinstalled the Windows iLO CHIF driver and reinstalled Windows from scratch incase the drivers were at fault, but this has not helped, still unable to flash / recover the firmware.

I have tried the following procedure to manually recover / flash iLO as described in these posts:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c02498232

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/iLO-4-Reset-Non-Responsive-iLO/td-p/6994298#.YFHlDJ37TDd

I can't get past the step to unload the upilo kernel module, rmmod hpilo returns:

ERROR: Module currently in use (or similar, it's at remote site and I can't see it now)

I've also tried to use modprobe -r to unload the module but this also doesn't work.

Any help would be really appreciated, I've got 2 dead iLOs (basically dead servers) and a couple of days to get one built! Thank you.

[AmRa]

Please refer below customer advisories and follow the workaround mentioned under resolution section.

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04996097

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning

https://support.hpe.com/hpesc/public/docDisplay?docId=a00048622en_us

[PC23]

Hi, thanks for your reply.

Step 1 in your advice is to upgrade the iLO, which I cannot do.

At present I can see that the iLO picks up an IP from DHCP, I can ping it, but I can't get the web interface to load. (POST reports IP V4 Unknown) Trying to access IP (pressing F10 at post) does nothing. It shows that F10 has been pressed but the nothing happens, no error and it doesn't enter IP.

Pressing F8 at post doesn't get me into iLO RBSU either.

Trying to reinstall IP using cp031302.exe does nothing, it starts extracting, a command window opens, then nothing.

Thanks.

[ksram]

Hi,

Please try to clear NVRAM referring : https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0128682

The link shared is for Gen9 Server however you may still follow the same.

Thank you

RamKS

[PC23]

Hi, thanks for your reply.

In this last link you have provided, for my case, I am just following step 1 only?

[ksram]

Hi,

Yes, that's correct.

Apologies for the delay in response.

Thank you

RamKS

[PC23]

Hi,

Followed the instructions, used switch 6 to clear NVRAM, got the onscreen message that this was done, followed instructions to shutdown and put switch back to normal and restart.

iLO problem remains, this had no effect.

[ksram]

Hi,

Thank you for trying the steps and sharing the update.

If all the steps are followed and if the issue is still the same, it may be due to the iLO Chip going faulty and may needs replacement. (System Board - since the iLO is integrated with the System Board)

Thank you

RamKS

[PC23]

Thanks for your reply. This is a shame. I can see that a lot of other people had the same issue or very similar.

On a couple of DL380p Gen8's from the same batch that I updated iLO first before doing the rest of the firmwares with SUM, the systems are working and are in production. All firmwares are up to date and iLO for the most part is working ok, however, both servers show the following warning on the iLO web interface login page:

'iLO Self-Test reports a problem with: Embedded Flash/SD-CARD. View details on Diagnostics page'

In the iLO overview I get 'iLO Health Degraded', system health is OK.

In diagnostics I have the following:

'Embedded Flash/SD-CARD Controller firmware revision 2.10.00 Embedded media initialization failed due to media write-verify test failure'

Can I get advice on this issue please.

Thanks.

[AmRa]

With reference to iLO Diagnostic tab will display the following error message:, please refer below customer advisory.

"Embedded media manager failed initialization"

HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04996097