iLO 1.96 upgrade failure

[Jungster]

Hi

I've been doing some digging into iLO firmware upgrade problems other people have been experiencing but none have really drawn any conclusions.

We have a number of HP Proliant DL380G4 servers running Windows Server 2003.

As part of the mitigation process of OpenSSL "Heartbleed" Vulnerability, we are going around upgrading the iLO firmware to 1.96 on all servers.

Downloaded the SoftPAQ cp023366.exe.

1. On our test rig,  we have 2 servers.On test server 1, we loaded the softpaq, this warned:

The software is not supported for this installation on this system.
Unable to communicate with the management processor.
Press Close to exit Setup.

Incidentally the server was rebooted only to indicate iLO was missing the BIOS bootup sequence, loading the Smart Array RAID module.

2. Test server 2 upgraded fine, with the BIOS bootup sequence showing iLO with the F8 option, IP address still intact, then loading the Smart Array RAID module.

3. Two live sites: the 3 servers at site one upgraded successfully. On the second site, of the 3 servers, only 1 upgraded successfully, but for the other 2 claimed to have upgraded successfully but after a reboot, iLO was missing the BIOS bootup sequence. The webui to the iLO interface displays the following message:

The iLO firmware is in a network flash recovery state.
Refer to the iLO network flash recovery under the troubleshooting section in the iLO users guide

The instruction was to attempted a manual upload having extracted the SoftPAQ to allow use of the firmware image file:

ilo196.bin

ftp ILO-IP

user flash

password recovery

bin

put ilo196.bin

this loads the 31 parts successfully and the ftp connection is dropped.

iLO was missing the BIOS bootup sequence.

I believe prior to the upgrade, the server's iLO firmware version were 1.91

After further research, I managed to revert the firmware to 1.94 using the ROMPAQ on test server 1:

Downloaded ROMPAQ version 1.94

Created floppy disk images

Disabled the iLO security (from the switch on the motherboard)

Boot server from floppies and follow prompts.

Shutdown server when complete and re-enable iLO security.

Booted server up, confirmed iLO was working once again.

Re-attempt firmware upgrade from SoftPAQ... results were the same as before... upgrade failed so re-applied ROMPAQ.

i also note the removal of the CMOS battery would result in the iLO configuration being lost (user accounts, licensing, IP addresses etc)

Questions:

Where does the SoftPAQ write it's log file?

When applying the SoftPAQ, is there any need to disable the iLO security from the switch on the motherboard?

Does the motherboard need replacing?

Please advise.

Thanks

Jung

[waaronb]

Can you try updating the firmware through the ILO web page itself, on that server where it says it's not supported on that system?

On the other ones that failed and they're in the recovery mode, I think you did the right steps, but if it still won't recover the ILO that way then you're probably going to need a motherboard replacement. I've had that happen once where ILO had a bad flash and simply couldn't be recovered no matter what.

The ILO settings are stored along with BIOS settings, so yeah, without the battery for long enough, the BIOS and ILO settings get wiped, same as if you did an NVRAM reset.

Windows 2003 is kind of old. Trying to install the online upgrade might have issues... HP doesn't really support the older OS's that well, but they made an exception for the Heartbleed patches.

I'd stick with doing the ILO updates through the ILO web site, or using the ILO scripting, because then you're sending the BIN directly and avoiding any bugginess of the upgrade utilities.