iLO 4 firmware update failure, stuck on "Checking HP Secure Digital Signature"

[Protonus]

Hello, I am trying to update the iLO 4 firmware on a ProLiant DL360p Gen8.   It's an ESXi host.  It's currently running iLO Firmware v1.5.1.  I am trying to update it to 2.61, same as our other servers.   I have done this update many times through the iLO Web Interface, by uploading the ilo4_261.bin file, etc.

A coworker of mine tried to update this particular server with this method, several weeks ago.  For some reason, this particular server has failed to update.  It is stuck on the screen of "Checking HP Secure Digital Signature" and at the bottom of the screen, indicates "Firmware Flash Progress: 1 %".  This never changes, it's been at this screen for weeks.  I see no option to cancel it, or try it again, etc. 

I have tried so many things to resolve this, here is a list:

• Tried resetting/restarting the iLO itself from the Diagnostics page.  Though no errors are generated, it does not seem to actually restart looking at the iLO Event Log.  I believe this is because it thinks it's updating the firmware still.
• This error / failure to update I'm seeing, is actually  mentioned in the HP iLO 4 user guide.  It says the resolution is to "hponcfg -r" from the command line.  I have done this from the /var/hp/tools/ install on ESXi on the server, and while it did succesfully reset settings to defaults, and wiped my user accounts out etc, the iLO still thinks that it's trying to update the firmware and the message is unchanged!
• I also tried rebooting the iLO using a RIBCL script, that invokes the "<RESET_RIB/>" command.  This too runs as expected, no errors, but does not seem to actually reset at all.   The firmware still thinks it's updating, and remains unchanged.
• I built a script to update the iLO firmware from the command line using hponcfg.  I verified the firmware location is valid (local, in the same folder) as are the user credentials.  This fails with the following error.  I can see this error in the RIBCL guide, and the error guide, but there is no explanation of what this error actually means or how to resolve it!  Very frustrating.  I am assuming that this error is just saying it's failing to flash, and perhaps that's for the same reason as why the WebUI didn't flash either? Here's the error in the full output:

/opt/hp/tools # ./hponcfg -f firmware.xml
HP Lights-Out Online Configuration utility


Version 4.4-0 (c) Hewlett-Packard Company, 2014
Firmware Revision = 1.51 Device type = iLO 4 Driver name = hpilo
iLO IP Address: 0.0.0.0
</-- ERROR :      STATUS= 0x005E
     MESSAGE= Open flash part failed. -->
Script failed
/opt/hp/tools #

---

I'm basically out of ideas.  The only thing left I can think to do, is to have the client go on site, and physically disconnect the server from power.  As this is a high availability host that is typically only remotely managed, I am trying to do whatever we can, to avoid that. 

Is there anything else that can be done to try and clear this error?  Thank you!

[Protonus]

Here's a screencap btw of the firmware update screen and where it's hung, been like this for weeks without any signs of progress, error, or ability to cancel.  Thank you in advance.

[ronissac]

Hi Protonus , 

Updating ILO firmware from 1.52 to 2.61 is not recommended. You will need to do an incremental update .

Also this issue has been fixed in ILO 2.10 firmware

I recommend the below incremental update

1.51 -> 2.10 -> 2.40 -> 2.50 -> 2.61

If you feel this was helpful please click the KUDOS! Thumbs below!

Regards

Ronny

[Protonus]

---

@ronissac wrote:

Updating ILO firmware from 1.52 to 2.61 is not recommended. You will need to do an incremental update .

Also this issue has been fixed in ILO 2.10 firmware

I recommend the below incremental update

1.51 -> 2.10 -> 2.40 -> 2.50 -> 2.61

---

Hello @ronissac - thank you for letting me know what version this bug is fixed in, and the proper builds to incremental update too.  We previously haven't had any issue going from any old version, directly to 2.61, so far it was unique to this site.

That all said though, unfortunatly this doesn't solve my original issue that I posted about.  How can I get this server out of the current state of "Checking HP Secure Digital Signature" / "Firmware Flash Progress: 1 %" ?  I would be happy to try the update to 2.10, but I see no way to cancel the current update attempt or reset the iLO that actually works?  Please review my original post to see what I've tried so far. 

Thank you very much!

[ronissac]

Kindly Power drain the server (shut down the server and unplugged the server from power source for a few mins) and then try to Reset ILO . 

[Protonus]

---

@ronissac wrote:

Kindly Power drain the server (shut down the server and unplugged the server from power source for a few mins) and then try to Reset ILO . 

---

Hello @ronissac - We had the customer go on site, and disconnect the server from power for a few minutes.  After some difficulties with starting it back up, and getting our VM's up, eventually the iLO did come back up as expected, and better still, it no longer said it was flashing firmware / checking security signature / stuck at 1%!

I then rebooted the iLO interface 2x, made sure each time it came up normally.  I then attempted to update it throuth the Web / Upload interface, to version 2.10 as you instructed.

And unfortunatly... it went right back to what it did before!   It's now been stuck at 1% again for many days, ugh.  We're right back where we started.  :-(

---

My only remaining thought, was to have the customer go back on site again, and power drain it again.   Once the iLO comes back up and it's no longer stuck in this state - my thought was to then try resetting the iLO to default settings via hponcfg over SSH on the ESXi host.  And then try and flash the firmware from the command line using hponcfg  from the server itself, rather than doing it over the network using the web interface.

Does this sound like what I should try next, or are we out of luck here and we need to leave it on firmware 1.51?  Or is there a different solution?

[Protonus]

---

@Protonus wrote:

---

@ronissac wrote:

Kindly Power drain the server (shut down the server and unplugged the server from power source for a few mins) and then try to Reset ILO . 

---

Hello @ronissac - We had the customer go on site, and disconnect the server from power for a few minutes.  After some difficulties with starting it back up, and getting our VM's up, eventually the iLO did come back up as expected, and better still, it no longer said it was flashing firmware / checking security signature / stuck at 1%!

I then rebooted the iLO interface 2x, made sure each time it came up normally.  I then attempted to update it throuth the Web / Upload interface, to version 2.10 as you instructed.

And unfortunatly... it went right back to what it did before!   It's now been stuck at 1% again for many days, ugh.  We're right back where we started.  :-(

---

My only remaining thought, was to have the customer go back on site again, and power drain it again.   Once the iLO comes back up and it's no longer stuck in this state - my thought was to then try resetting the iLO to default settings via hponcfg over SSH on the ESXi host.  And then try and flash the firmware from the command line using hponcfg  from the server itself, rather than doing it over the network using the web interface.

Does this sound like what I should try next, or are we out of luck here and we need to leave it on firmware 1.51?  Or is there a different solution?

---

Bump?  Do we need to replace this iLO board, or is there another option?  Anyway to hard reset it remotely using hponcfg etc, or tell it to cancel the firmware update in progress, other than power drains?   Is there a firmeware between 1.51 and 2.10 I should be trying instead?

[Mona_C]

Hi,

We are reposting the iLO 4 User Guide to the web with this troubleshootig tip:

iLO firmware update does not finish

When an iLO firmware update is initiated, the update does not start and one of the following symptoms occurs:

• The firmware update status remains at 1%.
• The message Checking HP Secure Digital Signature is displayed.

Solution 1

Cause: The iLO web interface stopped responding.

Action:

1. Refresh the browser window.
2. Retry the iLO firmware update.

Solution 2

Cause: The iLO firmware update process cannot make progress because other processes are consuming all available processing time.

Action:

1. Shut down the server OS.
2. Cycle the auxiliary power.
3. Retry the iLO firmware update before starting the server OS.

[Protonus]

---

@Mona_C wrote:

Hi,

We are reposting the iLO 4 User Guide to the web with this troubleshootig tip:

---

Thank you for the reply, but I have already done these steps, as well as read and reviewed the source guide you are posting.  It's referenced in this thread already.  Looking for a technical response to my prior two posts, thank you.

[Protonus]

Welp, I'm at my wits end with this server.  We tried power draining it again fully, then booted it up clean - verified that the iLO was in a normal state.

This time I pushed the bin file for intermediate version 2.10, up to the ESXi datastore directly, so the file was local.  I then tried to do the firmware update via hponcfg using the Update_Firmware.xml script.  While the script executed, it did eventually time out, because the iLO went right back to the "Checking HP Secure Digital Signature" status in the GUI, and it's once again stuck at 1% flash progress!  This is the same thing that's happened every time we try and update this iLO, no matter the version or update method.

I think the iLO itself needs to be replaced.  It's a ProLiant DL360p Gen8 - the iLO firwmare itself is on the motherboard, correct?  Would this require a new motherboard to resolve? 

[MDella]

Just as a side note, I have two servers that have this EXACT same problem (going from v1.50 to 2.70 and stuck). Note that both these servers have since been pulled out of production and stuck on a shelf. They have been powered down for a couple months and we just looked at them recently and still stuck in the iLO "Checking HP Digital Signature, please wait ..."

After a few days of searching around, its looking like the iLO is basically a brick and not likely to ever come back. We are putting these two servers up on EBay because of this problem. There is probably someone out there that doesn't care that the iLOs are bricked.

Marcos

BTW, ours are iLO4 on DL-360p (gen 8 servers).

[xrobau]

After doing some reverse engineering, this is an issue with the ANCIENT versions of ILO4. If the SYSTEM CLOCK is after 2015-ish, the certificate validation process will crash, and the upgrade process will hang at 1%.

The only way to fix this is to do a DIRECT FLASH. This means you need to turn the 'ILO Security' switch ON, and then boot the machine to Linux. Download the latest linux package. which is (at the time of writing) 2.81: https://downloads.hpe.com/pub/softlib2/software1/sc-linux-fw-ilo/p192122427/v213013/CP051872.scexe

Run 'bash CP051872.scexe --unpack=.' and then './flash_ilo4 --direct'

That disables all the verification and checks, and directly overwrites the ILO flash from the OS. I had to do that with a pair of BL460 G8's, and then I had to do a physical power cycle to remove the error light, but after that everything updated fine.