Betreff: Query: Oneview for vCenter: from Basline to VLCM

[pirx]

I think the issue is that you have installed Marvell's  "ESXCLI VMware Plug-in" only on some hosts. I ran into the same issue today. I tried to setup image based updates in a mixed cluster with Fujitsu and new HPE nodes (I doubt that this will work...). Anyhow, I noticed the same error you got and found no relevant driver package. Then I searched a bit a found that the difference between the old and new hosts is this package.

# esxcli software vib list |grep -i mrvl
qlogic-adapter-provider 1.7.23-10404389 MRVL VMwareAccepted 2022-09-27

So I gues you either have to remove this package or installed it where it is missing.

Edit: no sure if this is really the issue, as removing this package does not solve the problem for me. And yes, image based updates are broken and even VMware support highly recommended to me not to use it until I really have to.

[erki72]

hi @pirx 
Thank you for your hint.
Unfortunately I have no output with the command: esxcli software vib list |grep -i mrvl

I'm not sure if I should install it either. I think I have 4 hosts in the cluster, same model and all updated with oneview.

I don't like your last lines. Luckily I only switched to image based on one cluster. All others not yet. Although both variants are no longer working and VMware does not know why.

So, no news from me and I'll stay tuned with VMware support. If anyone out there has any advice, I'd be very grateful!

[T_1_6]

Seriously cannot beleive you are STILL making this so hard for yourself.

Select one revision back from latest for the HPE custom add-on, and get on with your life.

[pirx]

I'm not the OP but I've now the same issue and there is simply no package/version to go back to.

[T_1_6]

There is a large list in the custom add on section.

[erki72]

I would like to have it explained to me. I'm sorry if I'm too stupid to understand that..

HPE Customization for HPE Servers 703.0.0.11.1.0-25 > cant save


HPE Customization for HPE Servers 703.0.0.10.10.0-12 > Host not compatible with the image



HPE Customization for HPE Servers 703.0.0.10.9.1-5 > Host not compatible with the image (is way too far behind in my opinion anyway)


If I'm really that difficult to understand, I would be grateful if you could mark the screenshot where I can adjust something

[PatrickLong]

What is the ACTUAL FC HBA hardware model installed in your hosts? At least SOME of the confusion results from the fact that some of those QLogic FC chipsets were acquired by Marvell -and it gets even more murky when dealing with CNA adapters.  If it is anything from the QLogic chipset lineage you can get some detailed info by issuing the below command:

/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -a
Listing all system keys:
Key Value Instance: vmhba1/qlogic
Listing keys:
Name: ADAPTER
Type: string
value:
HPE SN1100Q 16Gb 2p FC HBA for SN1100Q:
FC Firmware Version: 9.09.00 (d0d5), Driver version 5.3.2.0

<followed by all the information you'd ever want to know about this hba and driver parameters>

Getting the driver vendor and version details via esxcli requires knowing either the vib name (in this case qlnativefc) or the three character case-sensitive vendor code (MVL):

esxcli software vib list |grep MVL
qlnativefc 5.3.2.0-1OEM.703.0.0.18644231 MVL VMwareCertified 2022-12-20

Maybe not so helpful to your immediate issue, but good commands to tuck away for future use.

[erki72]

Listing all system keys:
Key Value Instance: vmnic3/QLogic
Listing keys:
Name: rxrs
Type: integer
value: 64
Name: txrs
Type: integer
value: 64
Name: noq
Type: integer
value: 18446744073709551615
Name: dll
Type: integer
value: 0
Name: hwvlan
Type: integer
value: 1
Name: tpa
Type: integer
value: 1
Name: led
Type: integer
value: 2
Key Value Instance: vmnic2/QLogic
Listing keys:
Name: rxrs
Type: integer
value: 64
Name: txrs
Type: integer
value: 64
Name: noq
Type: integer
value: 18446744073709551615
Name: dll
Type: integer
value: 0
Name: hwvlan
Type: integer
value: 1
Name: tpa
Type: integer
value: 1
Name: led
Type: integer
value: 2
Key Value Instance: vmnic1/QLogic
Listing keys:
Name: rxrs
Type: integer
value: 64
Name: txrs
Type: integer
value: 64
Name: noq
Type: integer
value: 18446744073709551615
Name: dll
Type: integer
value: 0
Name: hwvlan
Type: integer
value: 1
Name: tpa
Type: integer
value: 1
Name: led
Type: integer
value: 2
Key Value Instance: vmnic0/QLogic
Listing keys:
Name: rxrs
Type: integer
value: 64
Name: txrs
Type: integer
value: 64
Name: noq
Type: integer
value: 18446744073709551615
Name: dll
Type: integer
value: 0
Name: hwvlan
Type: integer
value: 1
Name: tpa
Type: integer
value: 1
Name: led
Type: integer
value: 2

[:~] esxcli software vib list |grep MVL
qlnativefc 5.1.68.0-1OEM.703.0.0.18644231 MVL VMwareCertified 2022-10-31
[:~]

[pirx]

The MRVL package is also puzzling me. The HPE Upgrade Component is simply broken, the right package version would be 1.4.1 - that's what is installed with latest HPE ESXi ISO images and is in the HPE recipie. But... the 900 version is part of VMware's Non-Critical default baseline (but the package is from HPE). So if you patch your server with good old patch baseline incl non critical patches, the 900 version will be installed. And all other versions of upgrade component will be outdated. 

So if you still have 1.4.1 installed, overwrite version 900 with 1.4.1 in add component section, so that it never installes that forsaken version. But if you already have replaced 1.4.1 with 900 and get now the warning that you cannnot downgrade from 900... than replace whatever should be installed with version 900. Or unistall 900 manually and replace it with 1.4.1.  I've no idea what this component is used for and what the difference between the versions is. And HPE problably doesn't too, as I've opened a case for this and did not get any feedback yet.

This is all completely broken. And never try to setup image based backup with a mixed vendor cluster...

hpe-upgrade-component--1.4.1-1 and 900.0.2-1 - whe... - Hewlett Packard Enterprise Community

[erki72]

Yes, hpe-upgrade 900.0.2-1OEM.700.1.0.15843807 HPE PartnerSupported 2022-11-24 is installed on the Hosts.

But now it's getting exciting. I can't uninstall the part on all hosts at the same time. I try to save on one and then the image...

[T_1_6]

Hosts in this cluster are managed collectively. This image below will be applied to all hosts in this cluster.

ESXi Version 7.0 U3i - 20842708

Vendor Addon  HPE Customization for HPE Servers 703.0.0.10.10.0-12

Firmware and Drivers Addon  Gen10 Service Pack for ProLiant SPP-2022.09.01.00.703

Components  No additional components Show details

Works for me.

[pirx]

You installed your hosts with lasted ISO VMware-ESXi-7.0.3-20842708-HPE-703.0.0.11.1.0.25-Dec2022.iso from HPE from December? Because this comes with HPE-703.0.0.11.1.0.25-Dec2022-Addon-depot.zip and if you chose  703.0.0.10.10.0-12 is guess it will not work so straight forward.

[T_1_6]

Not installed, that was just the latest upgrade.

[pirx]

Which HPE Upgrade Components version is installed? 

[erki72]

No, but I still patched them with the update manager and the dynamic baselines.
Upgrade Baseline bevor patching in Update Manager:

VMware ESXi 7.0.3 Update 3, HPE-Custom-AddOn_703.0.0.10.10.0-12, 06/30/2022, 2:00:00 AM (created in vCenter)

Can I uninstall the piece of software on all hosts? Respectively what is the content in the component?
hpe-upgrade 900.0.2-1OEM.700.1.0.15843807

[erki72]

This one:

[:~] esxcli software vib list |grep hpe
hpe-upgrade 900.0.2-1OEM.700.1.0.15843807 HPE PartnerSupported 2022-11-24
[:~] esxcli software vib remove -n hpe-upgrade

[T_1_6]

No idea why you are using HPE "upgrade".

Never heard of it, or used it.

[PatrickLong]

Ah - OK so now we are getting somewhere - again this is probably not helpful in terms of your original question but hopefully this is related information that can be helpful to you.  You are NOT in fact using a native FC HBA which would use the QLogic Native Fibre Channel HBA Driver (qlnativefc) so there is no point in trying to upgrade or downgrade that component..  Your host has Marvell FastLinq 41000-series Adapter model QL4113HLCU adapters, which is in fact a CNA.  This is the murky water I mentioned earlier with who owns what chipset vs how the driver is branded, etc.  This is a Marvell-owned chipset that uses a QLogic-branded driver. The Marvell FastLinq 41000-series Adapter uses the following drivers under ESXi:

esxcli software vib list |grep qe
qedentv 3.70.7.0-1OEM.700.1.0.15843807 QLC VMwareCertified 2022-12-20  (this is the ethernet component driver)
qedf 2.30.8.3-1OEM.700.1.0.15843807 QLC VMwareCertified 2022-12-20 (this is the fcoe component driver)
qedi 2.30.7.2-1OEM.700.1.0.15843807 QLC VMwareCertified 2022-12-20 (this is the iSCSI component driver)
qedrntv 3.70.7.0-1OEM.700.1.0.15843807 QLC VMwareCertified 2022-12-20 (this is the RoCE component driver)

Note these drivers are NOT MRV Marvell-branded - they are still listed as Vendor QLC QLogic, and they are delivered as part of the component package MRVL-E4-CNA-Driver-Bundle_xxxxxxxxxxxxx.zip  << This is the component that you want to focus on for your server, not QLogic qlnative FC drivers.

HPE Support Site shows latest HPE-SUPPORTED version available for download is MRVL-E4-CNA-Driver-Bundle_5.0.339.0-1OEM.700.1.0.15843807_20031634.zip, available as cp050967.zip here.  HPE calls this driver Version: 2022.09.01 (28 Sep 2022) - just because they like to further confuse things and pretend these drivers are THEIR drivers by layering on their own cpxxxxxxx.zip filename (which is simply a re-zipped up version of the manufacturer-provided zipped driver) and giving it a name like 2022.09.01, that correlates to absolutely nothing in the actual driver documentation, file versions, or release date  I WOULD USE THESE.

However, I will note that VMware has a somewhat newer version of this driver component set available ,  MRVL-E4-CNA-Driver-Bundle_5.0.345.0-1OEM.700.1.0.15843807_20429303-package.zip (2022-11-02) which includes qedentv-3.70.7.0, qedf-2.30.8.3, qedi-2.30.7.2, and qedrntv-3.70.7.0..  NOTE, this newer version is not shown on the compatibility list for that hardware, which just means that the hardware has not been officially qualified on that driver version (but in all liklihood would work just fine)  If this is an environment that needs support, I would stick to the supported version from HPE shown above.

Also, just note per your screenshot his server has an A42 BIOS version of 2.60 (22 Aug 2022) which is the latest version included on the released SPP P52581_001_gen10spp-2022.09.01.00-SPP2022090100.2022_0930.1.iso, but the latest version is v.264 (1 Dec 2022), rated "Recommended" located here .  I would upgrade this manually via iLoREST and reboot the server if possible.since it has the latest reference code for the AMD processor(s) in this system.

[pirx]

No idea why you are using HPE "upgrade".

This  comes with every custom HPE ESXi ISO image, if you use this for your ESXi install, you have this package.

Fresh install with 2022-12 HPE image.

[root@sdev8023:~] esxcli software vib list |grep -i upgr
hpe-upgrade 1.4.1-1OEM.700.1.0.15843807 HPE PartnerSupported 2023-01-12

Problem is that a version 900 of this package is in the no-critical patch baseline repository. Which is simply broken.

[erki72]

Wow @PatrickLong !
It's amazing how willing you are to help me, many thanks. Your description is fully accurate and it agrees.
At the section with the test "I WOULD USE THESE.", do you mean the cp050967.zip here or did you want to refer to another link? Sorry if I'm asking stupid. I uninstalled the drivers and then downloaded and installed using the link. The output showed no change from before.

# Uninstall
esxcli software vib remove -n qedentv -f
esxcli software vib remove -n qedf -f
esxcli software vib remove -n qedi -f
esxcli software vib remove -n qedrntv -f
Reboot

# Install
esxcli software vib install -d /tmp/MRVL-E4-CNA-Driver-Bundle_5.0.339.0-1OEM.700.1.0.15843807_20031634.zip
Reboot

I'll do that on all 4 hosts.

Another question regarding the BIOS update.
If I upgrade it again manually, won't I get a similar problem with the image based update from Vmware vLCM? Or is that already to "exclude"?

[erki72]

What I find exciting now:
After uninstalling package "hpe-upgrade 900.0.2-1OEM.700.1.0.15843807 HPE PartnerSupported 2022-11-24" on a host, the status changes in the vCenter. I could probably remediate the host (don't know if it really works).

[Beasleyiv]

So how did this work out?  I'm facing the same issue.  I'd imagine that if we can simply remove the 900 vib and then remediate all should be good.  

[erki72]

You can try. It may be that if you do this once for all hosts, then everything will be fine.

Unfortunately, I have the next problem and I've been fighting with support for weeks.
However, I have the following combination:
HPE Oneview (with Server Profiles) > Oneview for vCenter > vCenter 7.0.3

[pirx]

I've a case open at HPE for months regarding this upgrade-component version 900.x issue. It's very painful and time consuming to make them understand, support is not even able to reproduce the issue as they are not able to setup a ESXi with vCenter connection + update baselines in their lab. If you have the same issue please give the case number 5370396895 as reference. Only if customers created cases and push this, it will eventually be solved. 

[TQ12]

So the solution would be to uninstall that 900 hpe component manually on all affected ESXi ?
In my case that ESXi7 component does not let me upgrade my ESXis to ESXi8. 
The error message is that a "downgrade" of that component is not supported.