topic Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state in HPE OneView

OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 09 Aug 2023 03:58:37 GMT

Sorry for spamming this forum but even after an 1h calll with an HPE OneView pre sales engineer, this could not clarified, maybe @ChrisLynch can help. A few months back I got information that ILO Amp will probably be discontinued but OV4VC will get the ability to update firmware via VMware vLCM.

Document - HPE OneView for VMware vCenter 11.3.0 Release Notes | HPE Support

Ability to do firmware update using vSphere Lifecycle Manager without a server profile for both monitored and managed servers

The release notes state that this should now be possible. But it's not working for me. Current OV4VC user guide also contains this info but also a lot of references to server profiles and server profile templates.

Document - HPE OneView for VMware vCenter 11.3.0 User Guide | HPE Support

NOTE: HPE OneView for VMware vCenter 11.2.0 and later extends the VMware vSphere Lifecycle Manager support for
both managed and monitored server hardware in HPE OneView.
Server Profile Template settings
To perform the firmware upgrade using VMware vSphere Lifecycle Manager, use the following Server Profile Template settings:
The best practice is to create a dedicated Server Profile Template for a cluster and set the baseline in Server Profile Template before
triggering the update from VMware vSphere Lifecycle Manager.
...
HPE OneView Hardware Support Manager makes the changes to the Server Profile if the Server Profile exists and does not change the
Server Profile Template during the VMware vSphere Lifecycle Manager process.

I don't see how this would work. If I add a server as Monitored (where I can not create any server profile) I get below error. If I change the servers state to Managed and create a server profile, it works (more or less).
Before I put more an more time into this, can someone please confirm that vLCM integration for fw updates is working in Monitor mode. And what extra steps are necessary because that is just not covered in the users guide.

INFO:HsmServiceLogger:Retrieved information of task 6160951e-1bc3-4159-91b6-391d329f6821: {"action":"POST_IMAGE_UPDATE","description":"PostImage update for host-3047462","estimatedTimeRemaining":0,"hosts":["host-3047462"],"id":"6160951e-1bc3-4159-91b6-391d329f6821","messages":["\"Error occurred while updating the firmware baseline in server profile.\""],"operationStatusCode":500,"progress":100,"startTime":"2023-08-03 09:20:43.395074","status":"FAILED"}

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

ChrisLynch — Wed, 09 Aug 2023 00:45:40 GMT

This is supported, but you need to ensure your HPE OneView appliance is at 8.3 or newer, and using the OV4VC 11.4 or newer appliance.

Feel free to PM me your email address, and I can take a number of your issues offline and direct to help you get on the right path.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 09 Aug 2023 05:35:39 GMT

@ChrisLynch

As far as I can see, I'm on latest versions for OV and OV4VC, there is no OV4VC version 11.4. I'll send you a PM - thanks for that offer!

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Mon, 14 Aug 2023 13:30:47 GMT

In the meantime I created case 5375419471. But as we don't have OV licenses yes I had to create a hardware case. Sadly this seems to go into the wrong direction. I had onother case open before and have already reset VUM DB multiplte times. I stated that this is not an isolated issue for one server or ILO and that I had resetted ILO already.

For me the issue is somewhere in OV, either I'm missing an important configuration step - or it is not possible to do vLCM updates in "Monitored" state, which would match current documentation for vLCM updates in OV4VC 11.3.

Make sure all the needed backup files are created and available
Reset VUM database in vCenter > https://kb.vmware.com/s/article/2147284
Access iLO GUI of server CZ...
Click System Information in the navigation tree
Click the Device Inventory tab.
Click Discovery. Disable the MCTP and wait for two minutes.
Perform an MCTP factory reset.
Enable the MCTP. The device should be displayed again.
Reset iLO 5.
Reboot the OS.
Retry the update on CZ... and share the result

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 16 Aug 2023 14:43:59 GMT

Interestion, this week OV4VC 11.4 was released., again with vLCM mentioned in release notes. Let's see if it works this time.

HPE OneView for VMware vCenter (OV4VC)
Current release - 11.4 (August 2023)
New features include:

vLCM vSphere Firmware Staging Support in OV4VC enables greater flexibility in timing of your firmware updates across your infrastructure. This feature reduces the time that ESXi hosts spend in maintenance mode during firmware updates.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

ChrisLynch — Wed, 16 Aug 2023 15:29:52 GMT

The OV4VC 11.3 release added support for HPE OneView Standard servers to update firmware using an SPP. This feature was specifically added to both OneView 8.2 (API first) and 8.3 (UI) natively (which is managed at the server hardware level and does not provide compliance tracking; that's reserved for OneView Advanced) and OV4VC HSM provider to help iLO Amp customers migrate to HPE OneView. We do have other customers using this feature successfully, in both modes OneView Standard and OneView Advanced.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 16 Aug 2023 17:13:29 GMT

Ok, I just hoped supports new action plan was not just trial an error.

I’ve just received a new Plan of action which i would like to share with you, please check below:
Update OV4VC to the recently released version 11.4 and retry the update
SW link > https://myenterpriselicense.hpe.com/cwp-ui/product-details/Z7500-63235/-/sw_free

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 16 Aug 2023 20:34:31 GMT

@ChrisLynch

After udpate to 11.4 my first update in Monitored mode with vLCM was now successful. So indeed 11.4 did the trick, not sure how other customers did it with < 11.3. No more "Error occurred while updating the firmware baseline in server profile" errors in the logs.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

ChrisLynch — Wed, 16 Aug 2023 20:18:17 GMT

I'm glad you have this issue resolved. I'll have to follow up with my R&D team on this finding.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Thu, 17 Aug 2023 08:42:35 GMT

One thing that seems to have changed after switching from ILO Amplifier to OV is the way that firmwares are updated. Before, only the missing, non-compliant fw were flashed. Now - even though vLCM shows only the ILO fw as non-compliant - all firmwares are flashed. Install set contains everything. This takes very long time and I don't understand why it is done this way. But maybe I don't understand how this works. I see that the "force install" option is used in OV task logs - which explains why it is done this way - but I don't see where this is configued, or if this hard coded. In Managed mode this would be set in the server profile, but this does not exist in Monitored mode.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Thu, 17 Aug 2023 10:22:13 GMT

Strange, the firware update state after vLCM update is not complete. In below example the host was compliant regarding firmwares but the "Firmware and Drivers AddOn" Host Version was "None". So everything was udapted - but ok, possibly because the "Firmware and Drivers AddOn" was not yet set.

So a complete install set was created. But the last 2 firmwares were not installed. I guess the reboot in UEFI mode was not triggered - server is now compliant in vLCM and OV. This seems kind of odd.

Those 2 firmwares will be installed next reboot, but it is unexpected and not visible in vLCM or anywere elese, only ILO shows this incomplete state. Which is basicially true, as the same firmware is already installed, only because of "force install" mode it got reinstalled (I guess) - but on the other hand "Update components lower than baseline" is a bit opposit to what I see.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Fri, 18 Aug 2023 15:09:40 GMT

I had a session with a HPE consultant, he confirms my findings. We are wainting on next steps by HPE. But it's obvious that the process/workflow/interaction between OV/OV4VC and vLCM is not stable. Depending on the situation, the results are unexpected and important checks are not done.

Situation:

- a host with some VMware packages missing was updated

- all firmware versions were already compliant

- but "Firmware and Drivers AddOn" state was still "None"

Questions:

- why does vLCM/HSM update all firmwares with "force install" option, in Monitored mode I can't see any option to modify this option? All firmwares are up to date. If only the "Firmware and Drivers AddOn" image is "none" but all firmwares are up to date, why not set Hosts Version = Image Version? And why is this done over and over from scratch in case of just one update failing?

- why is the final reboot not triggered? OV clearly states the the host must be rebooted to activate some firmwares. But this seems not to be returned to HSM/vLCM or a final check is missing.

vLCM/sut/OV uploaded all firmwares to ILO and installed all firmwares again with force install optione (why is that?)

After that compliance was checked and server was ok.

vLCM update result:

Remediation completed successfully

Completed 08/18/2023, 15:52:49

But reality was differerent, the final reboot to activate firmwares that gets flashed in UEFI was missing.

Checking ESXi host tasks shows that host was only rebooted once (15:25), but there was one more reboot necessary after the second "Update firmware" task (15:31).

According to vLCM everything was done and server was put out of maintenance at 15:52. But the 2 firmwares were missing, a manual reboot confirmed this.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 06 Sep 2023 06:03:43 GMT

I did not get any details yet, but at least that those are known issues and will be fixed in a later version. So I guess it's time to put away OV/OV4VC for good as this was the second POC we tried and time is limited.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Wed, 18 Oct 2023 08:12:40 GMT

I would like to inform you that the case has been accepted by Level 3, the issue seems to be a bug, because there is no option to prevent “force install”.

Just a few weeks/months later.... I get confirmation that this is probalby a bug. It would be reallly nice if HPE would start testing their SW before throwing it at customers. I mean, I noticed this behaviour at the first day I tested updates for servers in monitored state (no server profile).

And the feature of updating servers in monitored state also did not work for months until OV4VC 11.4 was released.

Re: OV4VC 11.3, OV 8.4, vLCM, updates in "Monitored" state

pirx — Tue, 14 Nov 2023 07:40:17 GMT

The issue with the “force install”FW on monitored severs is expected to be fixed in OV4VC 11.5..

I'm crawling from version to version.... If I'd only knew which version is the one that is finally working, I would put OV beside and then continue.