iLO 2.5 upgrade to 2.55 issues

[KarolisS]

Hello,

I have upgraded iLO 2.5 to 2.55 and now I have issues with 30+ servers. iLO configuration got cleared (connection to IRS, iLO advanced license removed, Directory Server Settings, Directory Groups, Local Users, Administrator user password). The Integrated Management Log is empty.

Configuration got cleared only for Gen8 servers but not for all.

UUID, Server Serial Number, System ROM, System ROM Date, Backup System ROM now is shown as Unknown. iLO reset does not resolves an issue.

System Information gives almost no info about servers health.

Can anyone please advise what should I do next? How to fix this mess...

[Torsten.]

It is a mess.

if you reboot the server or better pull the plugs, ILO will work again, but the settings are lost until you reconfigure.

[KarolisS]

you are right, after reboot iLO works, but settings are lost.
I was planing to upgrade 500+ more servers, but after this issue I will definetly won't going to do it.

Do you know why upgrade for some servers goes successfully and for some not?

[KenKrubsack]

KarolisS,

We ran into the same thing when we went to 2.54.  On the Gen8 servers only, we had 9 ILOs that pretty much lost their configs and needed to be re-done - which for us was about a 15% failure rate. There is a known flake that from your note here apparently wasn't resolved in 2.55 - but I suspect it is a pre-existing condition and the upgrade actally brings that issue to light.

This did not happen with the ILO4's on any of our Gen9 servers, and we have over 200 of them.

Ken

[Torsten.]

the related advisory is not 100% clear, but obviously the issue hits te partition table of the ILO NAND, so a reformat occurs. the nodes say it is solved with 2.54, but 2.55 still have the issue. or the issue is already there and is shown during upgrade

[EricL2324]

For what it's worth...I've been upgrading multiple Gen 8/9/10 servers from 2.50 to 2.55 without any issues or configuration loss.

[Ken Krubsack]

Torsten,

The people I was working with gave me the impression the second half of your statement is the situation - the issue is already there and the upgrade brings it to light.

Ken

[MikeG15]

How many people are experiencing this serious issue with HP iLO 2.55? Is there an expected fix coming for this firmware? I don't think we can upgrade until this latest problem is fully resolved.

[MikeG15]

Looks like there is also a known issue with ESX and iLo 2.54/2.55. It totally kills ESX: https://robbeekmans.net/vmware/howto-workaround-fixing-hp-hosts-stuck-reboot-ilo-upgrade-showing-lost-connectivity/

[Torsten.]

i think this is a different problem with sd cards only. the sd-card is "behind" ilo, if ilo reboots during update, the sd+card is unavailable during this reboot.

[Ken Krubsack]

Torsten,

Yeah, we'd never see that particular issue as we have VM installed on local disk rather than the SD cards.  We had a few on SD, but as soon as you install a patch or two you're out of space on the SD cards.

Ken

[tux1431]

I have upgraded 1.20 to 2.55 and all good no issue. 

[smokin901]

Hi,

Please look at following customer advisory

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00026427en_us