Firmware installation fails

[chypsa]

Hello all,

since yesterday, we are experiencing an odd issue. We have a 2-node cluster with 2x ProLiant DL325 Gen10 Plus v2., managed via OneView Yesterday, we installed firmware updates on N1 and rebooted with no issues. After installing all of the updates on N2, we noticed there was one remaining and in error state.

Since then, we have not been able to install it using:

• SPP and SUM
• HPE OneView
• Manually by downloading the FW package of the recommended version
• Manually by downloading the FW package of the same version (reinstall)
• Manually by downloading the FW package of the latest version.

We have tried:

• Removing and readding the Baseline
• Removing and readding the Hardware
• Removing and readding the Server Profile
• Rebooting the machine several times
• Removing the server from OneView and installing manually via SUM and via iLO.

SUT reports:

--------------------
[ INVENTORY REPORT ]
--------------------

ComponentCountIniLOTaskQueue............: 1
SUTUpdateableComponents.................: 0
UEFIUpdateableComponents................: 0
iLOUpdateableComponents.................: 1
RebootRequiredAfterDeploy...............: yes

Component 1)

ComponentFileName.......................: HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg
ComponentName...........................: Firmware Package - HPE Gen11 Boot Controller NS204i-u and HPE G
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 1.2.14.1010
State...................................: Exception
Criticality.............................: Optional
UpdatableBy.............................: Bmc
PS C:\Windows\system32> sut /deploy
No action needed, firmware deployment request and iLO task queue both are empty.

When running a manual SUM update, we get the error:

Deployment error. Unable to download file(s).

When running the FW update via iLO, we get:

Firmware Update

No matching target was found for this update/upload.

Any suggestions may help, since we seem to be completely stuck and out of options.

[support_s]

System recommended content:

1. HPE ProLiant DL325 Gen10 Server - Software and Configuration Utilities

2. HPE ProLiant DL325 Gen10 Plus v2 server - Overview

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[TVVJ]

Hello,

You may attempt installing the firmware separately by downloading it from the download page. Please review steps on the tab "Installation Instructions" to install it.

Regards,

[chypsa]

That's a decent idea. I have already tried the iLO and SUM paths, as stated above. I had not yet tried iLORest, so this morning, I took a shot:

Current FW version (reinstall):

Discovering data...Done
Uploading firmware: HPE_NS204i_Gen10P_Gen11_1.2.14.1009_A.fwpkg
Successfully checked 'HPE_NS204i_Gen10P_Gen11_1.2.14.1009_A.fwpkg'.
Uploading component HPE_NS204i_Gen10P_Gen11_1.2.14.1009_A.fwpkg.
[200] The operation completed successfully.
Component HPE_NS204i_Gen10P_Gen11_1.2.14.1009_A.fwpkg uploading successfully.
Waiting for iLO UpdateService to finish processing the component
ERROR : Error occurred while updating the firmware.

Recommended FW version:

Discovering data...Done
Uploading firmware: HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg
Successfully checked 'HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg'.
A component with the same name (HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg) has been found. Would you like to upload and overwrite this file? (y/n)y
Uploading component HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg.
[200] The operation completed successfully.
Component HPE_NS204i_Gen11_1.2.14.1010_A.fwpkg uploading successfully.
Waiting for iLO UpdateService to finish processing the component
ERROR : Error occurred while updating the firmware.

Latest FW version:

Discovering data...Done
Uploading firmware: HPE_NS204i_Gen10P_Gen11_1.2.14.1013_A.fwpkg
Successfully checked 'HPE_NS204i_Gen10P_Gen11_1.2.14.1013_A.fwpkg'.
A component with the same name (HPE_NS204i_Gen10P_Gen11_1.2.14.1013_A.fwpkg) has been found. Would you like to upload and overwrite this file? (y/n)y
Uploading component HPE_NS204i_Gen10P_Gen11_1.2.14.1013_A.fwpkg.
[200] The operation completed successfully.
Component HPE_NS204i_Gen10P_Gen11_1.2.14.1013_A.fwpkg uploading successfully.
Waiting for iLO UpdateService to finish processing the component
ERROR : Error occurred while updating the firmware.

[chypsa]

I have raised a support ticket. Another FW update on the same machine is also showing the same behaviour. The case nr. is: 5379649900.

[Sebasbin]

Hi,

I would suggest to power cycle the server and then reapply the server profile in OneView to update the firmware

[chypsa]

@Sebasbin 

This has already been done, several times in more than one way.

[Mr_Techie]

---

@chypsa wrote:

@Sebasbin 

This has already been done, several times in more than one way.

---

You can give a try to below instructions:

Confirm the server (N2) has proper network connectivity and it can reach the necessary repositories or download servers.

Verify that there are no firewall or proxy issues preventing the server from downloading the necessary files and check if any proxy settings are configured in the server or OneView that might be causing the issue.

Check the OneView configuration for both nodes. Ensure that the configuration for N2 is correct and matches that of N1 and check for any errors or discrepancies in the OneView logs related to the firmware update for N2.

Ensure that there is enough free space on the server (N2) to accommodate the firmware update files.

Check if there are any firmware dependencies or prerequisites for the update that might not have been met and ensure that the firmware versions on N2 are compatible with the updates you are trying to apply.

Manually download the firmware update file from the official HPE support website and attempt to manually apply the firmware update using the iLO interface, providing the locally downloaded file.

Verify that iLO settings on N2 are correct and match those on N1 and make sure that iLO is functional and accessible.

As a lat option, if the firmware update is critical and causing issues, consider reverting to the previous firmware version (if possible) to restore stability.

If you still facing the same issue, please reach out the HPE Support for further assistance.

[chypsa · Accepted solution]

For posterity:

The motherboard was the culprit.

I've contacted HPE support and they tried reslotting the Boot Controller to a different slot, after which it installed the firmware normally. However, after placing it back to the original slot (physical requirement), it started misbehaving again.

So, we changed the riser, which didn't help.

After that, we replaced the motherboard and now everything is working as expected.

One thing to note is that no one here suggested observing the Device Inventory (Unknown state on Boot Controller) and resetting MCTP settings to Factory Defaults, which can be of use in certain cases.

So, for those reading this sometimes in the future: you can also try that. If you reset MCTP to factory settings and your device is still in Unknown state, to me now, it seems likely that it's going to be a physical component issue. Obviously, might also be something else, but I'm just giving my experience.

Kind regards to all and thanks to those who tried to assist.

[Sunitha_Mod]

Hello @chypsa,

That's great! 

We are glad to know you were able to find the solution and we appreciate you for keeping us posted. We hope this helps the other users in the Community. 

[WimBerkhof]

It looks like the firmware upgrade of ns204i isn't flawless at all.

We now have two instances where the server had to be reset to 'factory defaults' using the dip switches.
One platform is DL360 Gen11, the other is BL480 Gen11.

I searched 'support.hpe.com' for any advisory on the issue, nothing found.
The firmware package isn't updated lately.