Host status is unknown: Ilo Amplifier 2.20 + vLCM + SPP-2023.03.00.00.703

[pirx]

This is now the third or forth time I see this after new SPP and AddOns packages are released. New 'HPE SPP Gen10 Service Pack for ProLiant SPP-2023.03.00.00.703' is out together with AddOn package 'HPE Customization for HPE Servers 703.0.0.11.3.0-5'. New SPP imported to ILO Amplifier, AddOn package to vCenter/vLCM. Resync started in vCenter.

Modified image to:

ESXi Version 7.0 U3l - 21424296

Vendor Addon HPE Customization for HPE Servers 703.0.0.11.3.0-5

Firmware and Drivers Addon Gen10 Service Pack for ProLiant SPP-2023.03.00.00.703

According to https://vibsdepot.hpe.com/customimages/Valid-vLCM-Combos.pdf this combination should work. But again I only get "Host status is unknown" after compliance check for the 3 hosts in that cluster. When I remove the SPP from the image the error disappears. vLCM and Ilo Amplifier was working before, with last AddOn package and bp-Convenience-download-2022.09-01.02-63ecb154be21b17921e3214f.iso.

I've no clue why there is now a internal server error. This worked before adding the new SPP, why does this break the scan of the hosts by ILO Amplifier? I can update the hosts from within ILO Amplifier with a new SPP baseline or create a firmware baseline report. It's just that vLCM <-> ILO Amplifier HSM is broken.

10.11.0.55 = Ilo Amplifier

Hosts are added to vCenter cluster configuration with their ILOs, they are in ILO Amplifier. Last times I backed up config, deployed ILO Amplifier again, restored config, imported SPP and it worked again. But I've no time to do this over and over again and again.

ERROR:HsmServiceLogger:["Failed to start scan for compliance for ['host-2429197', 'host-2429156', 'host-2496701']"]


NFO:HsmServiceLogger:Running command from C++: hosts scan
INFO:HsmServiceLogger:Running command hosts scan
INFO:HsmServiceLogger:Got operation arguments: _OptArgsHolder(host_id=None, package=None, release=None, task_id=None, version=None)
INFO:HsmServiceLogger:Initiating executor
INFO:HsmServiceLogger:Got network location: 10.11.0.55
INFO:HsmServiceLogger:Extracted hostname: 10.11.0.55, port: 443
INFO:HsmServiceLogger:Successfully initiated executor
/usr/lib/python3.7/site-packages/urllib3/connectionpool.py:988: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.11.0.55'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
ERROR:HsmServiceLogger:get_host_inventory failed with InternalServerError
ERROR:HsmServiceLogger:['Failed to start inventory for host-2429197']
INFO:HsmServiceLogger:Got http response status code: 500
ERROR:HsmServiceLogger:Error code: 500
INFO:HsmServiceLogger:Hsm service result: {'output': 'null', 'error': {'errorCode': 500, 'command': ('hosts', 'get'), 'input': '{"requestContext": null, "optArguments": {"host_id": "host-2429197", "package": "Gen10 Service Pack for ProLiant ", "version": "SPP-2023.03.00.00.703"}}', 'args': {'status': 500, 'message': ['Failed to start inventory for host-2429197']}}}
ERROR:HsmServiceLogger:scan_host failed with InternalServerError
ERROR:HsmServiceLogger:["Failed to start scan for compliance for ['host-2429197', 'host-2496701', 'host-2429156']"]
INFO:HsmServiceLogger:Got http response status code: 500
ERROR:HsmServiceLogger:Error code: 500
INFO:HsmServiceLogger:Hsm service result: {'output': 'null', 'error': {'errorCode': 500, 'command': ('hosts', 'scan'), 'input': '{"requestContext": null, "requestBody": {"action": "SCAN", "targetHosts": ["host-2429197", "host-2496701", "host-2429156"], "targetImage": {"addOnName": "HPE-Custom-AddOn", "addOnVersion": "703.0.0.11.3.0-5", "baseImageVersion": "7.0.3-0.85.21424296", "swComponents": {"BCM-vmware-storcli64": "007.2207.0000.0000-01", "Broadcom-ELX-IMA-plugin": "12.0.1200.0-4vmw.703.0.20.19193900", "Broadcom-ELX-brcmfcoe": "12.0.1500.2-3vmw.703.0.20.19193900", "Broadcom-ELX-lpfc": "14.0.622.2-1OEM.700.1.0.15843807", "Broadcom-bnxt-Net-RoCE": "224.0.249.0-1OEM.700.1.0.15843807", "Broadcom-elxiscsi": "12.0.1200.0-9vmw.703.0.20.19193900", "Broadcom-elxnet": "12.0.1250.0-5vmw.703.0.20.19193900", "Broadcom-lpnic": "11.4.62.0-1vmw.703.0.20.19193900", "Broadcom-lsi-mr3": "7.722.02.00-1OEM.700.1.0.15843807", "Broadcom-lsi-msgpt2": "20.00.06.00-4vmw.703.0.20.19193900", "Broadcom-lsi-msgpt3": "17.00.12.00-1vmw.703.0.20.19193900", "Broadcom-lsi-msgpt35": "23.00.00.00-1OEM.700.1.0.15843807", "Broadcom-lsiv2-drivers-plugin": "1.0.0-12vmw.703.0.50.20036589", "Broadcom-ntg3": "4.1.9.0-4vmw.703.0.85.21424296", "Cisco-nenic": "1.0.45.0-1OEM.700.1.0.15843807", "Cisco-nfnic": "5.0.0.37-1OEM.700.1.0.15843807", "ESXi": "7.0.3-0.85.21424296", "HPE-hpv2-hpsa-plugin": "1.0.0-3vmw.703.0.20.19193900", "HPE-nhpsa": "70.0051.0.100-4vmw.703.0.20.19193900", "Intel-NVMe-Vol-Mgmt-Dev-Plugin": "2.7.2173-1vmw.703.0.20.19193900", "Intel-SCU-rste": "2.0.2.0088-7vmw.703.0.20.19193900", "Intel-Volume-Mgmt-Device": "3.0.0.1038-1OEM.700.1.0.15843807", "Intel-i40en": "2.4.1.0-1OEM.700.1.0.15843807", "Intel-icen": "1.10.4.0-1OEM.700.1.0.15843807", "Intel-igbn": "1.10.2.0-1OEM.700.1.0.15843807", "Intel-irdman": "1.4.1.0-1OEM.700.1.0.15843807", "Intel-ixgben": "1.14.2.0-1OEM.700.1.0.15843807", "Intel-ne1000": "0.9.0-1vmw.703.0.50.20036589", "MRVL-Atlantic-Driver-Bundle": "1.0.3.0-8vmw.703.0.20.19193900", "MRVL-E3-Ethernet": "1.1.0.11-1vmw.703.0.20.19193900", "MRVL-E3-Ethernet-iSCSI-FCoE": "3.0.202.0-1OEM.700.1.0.15843807", "MRVL-E4-CNA-Driver-Bundle": "5.0.345.0-1OEM.700.1.0.15843807", "MRVL-QLogic-FC": "5.3.80.0-1OEM.703.0.0.18644231", "Mellanox-MFT-Tools": "4.23.0.1048-1OEM.703.0.0.18644231", "Mellanox-NATIVE-NMST": "4.14.3.3-1OEM.700.1.0.15525992", "Mellanox-nmlx4": "3.19.70.1-1OEM.670.0.0.8169922", "Mellanox-nmlx5": "4.22.73.1004-1OEM.703.0.0.18644231", "Microchip-smartpqi": "70.4380.0.108-1OEM.700.1.0.15843807", "Microchip-smartpqiv2-plugin": "1.0.0-8vmw.703.0.20.19193900", "Micron-mtip32xx-native": "3.9.8-1vmw.703.0.20.19193900", "Pensando-ionic-en": "18.0.0-1OEM.700.1.0.15843807", "Solarflare-NIC": "2.4.0.2010-6vmw.703.0.20.19193900", "VMware-NVMe-PCIe": "1.2.3.16-2vmw.703.0.85.21424296", "VMware-NVMeoF-RDMA": "1.0.3.5-1vmw.703.0.20.19193900", "VMware-NVMeoF-TCP": "1.0.0.1-1vmw.703.0.35.19482537", "VMware-VM-Tools": "12.2.0.21223074-21225287", "VMware-ahci": "2.0.11-1vmw.703.0.20.19193900", "VMware-iser": "1.1.0.1-1vmw.703.0.50.20036589", "VMware-nvme-pcie-plugin": "1.0.0-1vmw.703.0.20.19193900", "VMware-nvme-plugin": "1.2.0.44-1vmw.703.0.20.19193900", "VMware-nvmxnet3": "2.0.0.30-1vmw.703.0.20.19193900", "VMware-nvmxnet3-ens": "2.0.0.22-1vmw.703.0.20.19193900", "VMware-oem-dell-plugin": "1.0.0-1vmw.703.0.20.19193900", "VMware-oem-hp-plugin": "1.0.0-1vmw.703.0.20.19193900", "VMware-oem-lenovo-plugin": "1.0.0-1vmw.703.0.20.19193900", "VMware-pvscsi": "0.1-4vmw.703.0.20.19193900", "VMware-vmkata": "0.1-1vmw.703.0.20.19193900", "VMware-vmkfcoe": "1.0.0.2-1vmw.703.0.20.19193900", "VMware-vmkusb": "0.1-8vmw.703.0.85.21424296", "amsdComponent": "701.11.9.0.9-1", "amsdvComponent": "701.11.3.0.17-1", "esx-update": "7.0.3-0.85.21424296", "fc-enablement-component": "700.3.9.0.4-1", "hpe-upgrade-component": "1.4.1-1OEM.700.1.0.15843807", "ilo-driver": "700.10.8.0.6-1OEM.700.1.0.15843807", "ilorest-component": "700.4.0.0.0.32-15843807", "ssacli2-component": "6.15.11.0-7.0.0", "sutComponent": "701.4.1.0.8-0", "vnic-enablement-component": "700.2.10.10-1"}}, "targetPackage": "Gen10 Service Pack for ProLiant ", "targetPackageVersion": "SPP-2023.03.00.00.703"}}', 'args': {'status': 500, 'message': ["Failed to start scan for compliance for ['host-2429197', 'host-2496701', 'host-2429156']"]}}}

[Sunitha_Mod]

Hi, 

We have reported the issue to the support team and they will reach out to you. Please let us know the solution once the issue has been resolved. 

[FR_Wolfman]

Hi,

I have also issues with this combination of components (same versions also). First, I was getting an error about a conflict with ionic-en VIB. I removed it from one ESXi host to test, and now I get this unknown status error on the host.

If I remove the SPP baseline, there is no error and I am able to update the hosts with the latest ESXi build and HPE Add-on drivers.

The vLCM/HSM combination should help to update easily and seamlessly the servers to fully supported compatibilities, but each time there is a new release in the vLCM supported definitions, it seems to be a pain in the a$$ to get the things working smoothly.

I am trying to generate a custom SPP with only the firmwares (no drivers) to see if it can help.

[pirx]

The issue is in the couchdb.ini (value max_document_size). Once I doubled that value, ILO Amplifier started working again. To change the value you have to boot ILO Amp VM from a Linux ISO and do all the stuff you would do to access a VM and its filesystem that can't boot anymore. If you don't know how to do this, its probably better to wait until HPE finally releases a fix. Support knows about this.