Updating firmware with SUM from SPP for servers managed by OV

[pirx]

I've finally given up using OV for firmware updates. I spent 3 days this week trying update a single server with different methodes OV provides. Monitored mode, managed mode (all with vLCM), as well as direct from OV, also with Actions -> Update Firmware. 

Some of the failures might be related to the fact that we have server at remote kocations with a network latency >100ms (but no packet loss). I learned a few weeks ago from HPE consultants presenting OneView and Ops manager, the OV is only designed to work in local datacenter and that we would have to deploy OV instances everywhere around the world. Not possible.

So I tried latest SPP with SUM, copied it to a local server, added the server, started inventory, started update - firmware only.

All looks good for the first 4 updates, then the next update (Broadcom NX1 FW for VMware) waited in Pending state forever. sut is set to AutoDeploy. Server was added to OV as "monitored". Nothing I tried did unlock the state. I even tried to trigger deploy again from ESXi shell. I restarted the update from SUM, noch change, first update stuck in pending state.

Then I removed the server completely from OV, restarted update in SUM and no pending state anymore.

So for me it seems that SUM can not be used for servers that are manged my OV. Or ist there a configuration that removes OV as System Update Manager?

That would mean that we can't use OV for firmware updates (which is preatty clear for some time now), but we could also not use it for basic monitoring. Which makes it even more preposterous that we have to buy an OV or Ops manager license now with every server.

# sut /deploy
Operation Requestor is HPOneView and Request state is InstallFailed, not starting deployment until staged or installing or scheduled state is reached

# sut /status
System Update Manager...................: HPE OneView
Update Manager URL......................: https://xxxx
Deploy Type.............................: FirmwareOnly
Task Status.............................: Bios:N/A; LocalStorage:N/A; Deploy:InstallFailed
Staging Directory.......................: /tmp/stagingdirectory
Mode of Operation.......................: AutoDeploy
Polling Interval In Minutes.............: 5
Force Deploy............................: Yes
Optional Components.....................: None
EnableiLOQueuedUpdates..................: true
User Action Needed......................: If iSUT is running in AutoDeploy or AutoDeployReboot mode, no user action needed. Otherwise, use the deploy or deployreboot command to deploy the components.
iLO Security Mode.......................: Production
iLO CAC Smartcard Authentication........: Off
iLO CAC Strict Mode.....................: Off

--------------------
[ INVENTORY REPORT ]
--------------------

ComponentCountIniLOTaskQueue............: 9
SUTUpdateableComponents.................: 3
UEFIUpdateableComponents................: 1
iLOUpdateableComponents.................: 5
RebootRequiredAfterDeploy...............: yes

Component 1)

ComponentFileName.......................: HPE_UBM6_1.04_A.fwpkg
ComponentName...........................: Firmware Package - UBM6 Backplane PIC PLDM Firmware for Gen10/G
RebootRequired..........................: no
VersionIniLOTaskQueue...................: 1.04
State...................................: Complete
Criticality.............................: Optional
UpdatableBy.............................: Bmc

Component 2)

ComponentFileName.......................: ilo5_298.fwpkg
ComponentName...........................: Online ROM Flash Firmware Package - HPE Integrated Lights-Out 5
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 2.98
State...................................: Complete
Criticality.............................: Optional
UpdatableBy.............................: Bmc

Component 3)

ComponentFileName.......................: U46_1.80_07_05_2023.fwpkg
ComponentName...........................: ROM Flash Universal Firmware Package - HPE ProLiant DL360/DL380
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 1.80_07-05-2023
State...................................: Complete
Criticality.............................: Optional
UpdatableBy.............................: Bmc

Component 4)

ComponentFileName.......................: bcm226.1.107.0.pup.fwpkg
ComponentName...........................: Broadcom Firmware Package for BCM5741x adapters
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 226.1.107.0
State...................................: Complete
Criticality.............................: Optional
UpdatableBy.............................: Bmc

Component 5)

ComponentFileName.......................: HPE_NS204i_Gen11_1.2.14.1012_A.fwpkg
ComponentName...........................: Firmware Package - HPE Gen11 Boot Controller NS204i-u, NS204i-d
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 1.2.14.1012
State...................................: Complete
Criticality.............................: Optional
UpdatableBy.............................: Bmc

Component 6)

ComponentFileName.......................: CP058144.zip
ComponentName...........................: Broadcom NX1 FW for VMware
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 1.35.50
State...................................: Pending
Criticality.............................: Optional
UpdatableBy.............................: RuntimeAgent

Component 7)

ComponentFileName.......................: CP053486.zip
ComponentName...........................: Online HDD/SSD Flash Component for VMware ESXi  - EO000400PXDBQ
RebootRequired..........................: no
VersionIniLOTaskQueue...................: HPD3
State...................................: Pending
Criticality.............................: Optional
UpdatableBy.............................: RuntimeAgent

Component 8)

ComponentFileName.......................: CP056121.zip
ComponentName...........................: HPE Firmware Flash for QLogic Fibre Channel Host Bus Adapters f
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 2023.09.01
State...................................: Pending
Criticality.............................: Optional
UpdatableBy.............................: RuntimeAgent

Component 9)

ComponentFileName.......................: SPSGen10Plus_04.04.04.500.zip
ComponentName...........................: ROM Flash Firmware Package - Server Platform Services (SPS) Fir
RebootRequired..........................: yes
VersionIniLOTaskQueue...................: 04.04.04.500
State...................................: Pending
Criticality.............................: Optional
UpdatableBy.............................: Uefi

[ChrisLynch]

I'm sorry you continue to run into firmware deployment issues.  OneView is designed for datacenter management.  There are methods used that are not tolorant to high latency (yes, 100ms is high latency), and local instances of OneView should be deployed.  If that local instance is another datacenter.  If the servers you have deployed in other regions are ROBO (RemOte Branch Offices), trying to deploy OneView in those locations does not make sense.

You cannot use SUM to deploy firmware when the server is managed by OneView.  SUM is also not designed to deploy or manage firmware at scale.  We do have many other customers using OneView at scale to manage firmware.  If you have a support case, please DM me and I will help move it to a higher level support team.

Compute Ops Management could be looked as as an alterntive here, for your Gen10 and newer systems only.

[pirx]

I really would like to get in contact with a customer that uses OV to deploy firmware at scale. I asked this a while ago our local HPE contact but did not get any real feedback. I personally know a lot of people from other companies (including a car manufacturer) that tried it and failed like me - and I really tried hard. So it would really be nice to talk to someone using it with success (preferably in Germany). Especially with vLCM integration.

Last case I opened was 5375858533. But as the feedback and progess in general for such kind of cases is... not really existing... and only costs me a lot of time, I'll not open any more cases. My last ILO-Amp case was open for months and in the end I booted the appliance from and Linux ISO and fixed it by myself by changing one value (that support mentioned weeks ago in mail). Last feeedback from my local contact - not support - about my issues with vLCM was... they are known and a team at HPE is now trying to fix them to get it in an usable state. It wasn't even necessary to provide them details.

Opposit to OV, SUM was able to update the servers without issues once I've removed the server from OV. 

[ChrisLynch]

Opposit to OV, SUM was able to update the servers without issues once I've removed the server from OV. 

Without examining what happened by you providing an HPE OneView appliance support dump, I can't tell you what caused the issue to begin with.

I will look into the case you provided and see what it was about.

[pirx]

There is still 5375858533 open which was created in August... last feedback from some weeks ago in October.

I would like to inform you that the case has been accepted by Level 3, the issue seems to be a bug, because there is no option to prevent “force install”.

However, the colleagues are going to test this in a OV4VC environment with the assistance of Level 3.

Thats more a generic issue in vLCM + monitored mode. But you see how long it takes to get some feedback. I can create a new case for next server failing but I can't wait months with the updates. So once I get feedback months later, I've moved on and updated server in a different way.  If you are interested I can gather logs and open a case again. 

[ChrisLynch]

If you can, please do.  Then provide me with the support case so I can have someone assist it along with the Support Engineer assigned to the case.  Please collect a full appliance support dump, along with sut logs (if you were attempting to install firmware online).