OneView for VMware vCenter ov4vc not working after OneView 8.9 upgrade

[ndavis_cmm]

Since we upgraded to OneView 8.9 we see "host status is unknow" when trying to check the image compliance. The odd thing is everything seems to be communicating as far as iSUT is concerned. We can't seem to find anyting broken that is not allowing the compliance check to function. We have had this happen to two of our vCenters. Has anyone else had this issue? 

We have a case open with HPE, but are waiting on them to get back to us. 

Thank you,

Nate

[shuff]

The issue you're experiencing with the 'host status is unknown' error in ov4vc following an upgrade could be related to several factors. Common issues include stale or corrupt cache or cookies, internal server errors on vSphere Client, or problems with the host to iLO association marked by the 'AMS_NOT_INSTALLED' error. It's also possible that the vCenter is failing to fetch and list supported drivers from the Hardware Support Package, which can affect image compliance checks.

To troubleshoot these issues, you should first clear any stale or corrupt cache and cookies in your system. Check for any internal server errors on the vSphere Client and ensure that the AMS is correctly installed and functioning. Additionally, verify that the vCenter can successfully fetch and list all necessary drivers from the Hardware Support Package. If these steps do not resolve the issue, consider reviewing the specific error logs provided by the vSphere Client or the compliance check reports for more detailed diagnostics.

this doc might help: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sd00004012en_us&page=GUID-2C34A09F-9653-44A8-8964-1F55FA67CE1F.html

[support_s]

System recommended content:

1. HPE OneView for VMware vCenter Release Notes (9.2 and 9.2.1)

2. HPE OneView for VMware vCenter User Guide (9.1)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[ndavis_cmm]

@shuff Hello thank you for your reply.

The iSUT state and AMS state for the hosts are green. The iLO is reachable and I performed the steps to restart the AMS and then Reset the iLO, but that did not resovle the issue from the article you linked. 

I am not certain what "iLO Mapping" means if you can elaborate on that point? Not sure what I need to check there. 

Thanks, 

Nate

[ndavis_cmm]

This is the page I am referring to specifically https://support.hpe.com/hpesc/public/docDisplay?docId=sd00004012en_us&page=GUID-F56B101F-0273-44CF-84C1-8552B9217FF0.html

[Rajaram02]

Please share us the HPE case number 

[rkiik]

Hello,

Have you found any solution to this problem, I'm facing the same thing.

Regards,

Rene

[pirx]

Same here. Updated first from OV4VC 11.5 to 11.6 and all was working for a few days (8.80 OV), Then I updated OV 8.80 to 8.90, checked complicane and.... its broken (again). Host status is unknown. 

Something always breaks, either its OV4VC, OV or a new SPP. Its such a mess. Revered to 8.80 and hosts state is back to normal.

@shuff @ndavis_cmm 

Not sure how this KB should help...

After Image Compliance Check, some of the hosts in vCenter show the status as unknown | HPE OneView for VMware vCenter 11.5.0 User Guide

After Image Compliance Check, some of the hosts in vCenter show the status as unknown

Symptom

Some hosts in vCenter display a Host status is unknown message after the compliance check operation.

Solution 1

Action

If the host is not mapped to its iLO IP, then complete the host to iLO mapping operation.

Solution 2

Action

If the iLO of the host is unreachable, fix the issue and retry the compliance operation.

Solution 3

Action

If the AMS on the host was down, restart the AMS on the host, reset the iLO, and retry the scan operation after 10 minutes.

1. what is this mapping? How do I complete the mapping? The systems exist in OV and was working before. 
2. ILO is reachable like it was witj 8.80

3. AMS was/is not down

Not sure if this helps, but this is the part of the hsm log on vCenter

HsmService:2024-05-14 13:52:21,659[Dummy-41]hsmService:417 [INFO] Retrieved information of task 6d669a31-4313-4341-8604-123fab59c1ff: {"action":"GET_HOST_INVENTORY","description":"Firmware inventory details of host host-3405617","estimatedTimeRemaining":0,"hostInventoryList":[{"firmwareVersion":"1.57 Feb 27 2024","systemComponent":{"description":"SystemBMC","id":"827e7c5f-7bf0-4c90-9b33-aca3d0a450e0","name":"iLO 6","type":"EXTERNAL"}},{"firmwareVersion":"U63 v2.12 (12/13/2023)","systemComponent":{"description":"SystemRomActive","id":"5ab13b05-9b1e-4ade-8758-86ef02e62785","name":"System ROM","type":"BIOS"}},{"firmwareVersion":"10.3.0 Build 31","systemComponent":{"description":"PlatformDefinitionTable","id":"77836c92-c5ad-4f06-b95b-b6fede1b8448","name":"Intelligent Platform Abstraction Data","type":"EXTERNAL"}},{"firmwareVersion":"0x14","systemComponent":{"description":"SystemProgrammableLogicDevice","id":"10e87f90-7c41-4097-a21e-94c60fee9aa0","name":"System Programmable Logic Device","type":"EXTERNAL"}},{"firmwareVersion":"3.01","systemComponent":{"description":"PowerSupplies","id":"2f388cfd-b93e-4821-9997-3a0e57ea5da8","name":"Power Supply Firmware","type":"EXTERNAL"}},{"firmwareVersion":"3.01","systemComponent":{"description":"PowerSupplies","id":"f8244d14-bbbe-4881-92c6-afd09c5ea122","name":"Power Supply Firmware","type":"EXTERNAL"}},{"firmwareVersion":"6.1.4.05.0","systemComponent":{"description":"SPSFirmwareVersionData","id":"31d1743d-bf72-4fe2-b89b-4bcc920d5bd9","name":"Server Platform Services (SPS) Firmware","type":"EXTERNAL"}},{"firmwareVersion":"U63 v2.12 (12/13/2023)","systemComponent":{"description":"SystemRomBackup","id":"cc18c230-d097-47e0-b921-69ae5eb3fa2e","name":"Redundant System ROM","type":"EXTERNAL"}},{"firmwareVersion":"4.20.11","systemComponent":{"description":"Intelligent Provisioning","id":"7425c1b0-4ce7-4f5c-88ce-dfc0aa988336","name":"Intelligent Provisioning","type":"EXTERNAL"}},{"firmwareVersion":"0x1E","systemComponent":{"description":"IntelCFR","id":"1d35a2d2-c3ef-4a19-b65a-e3c71809c513","name":"Processor 1 S3M Firmware","type":"EXTERNAL"}},{"firmwareVersion":"0x3B000020","systemComponent":{"description":"IntelCFR","id":"74c25191-c6e9-4918-b0fe-51a3606f60bd","name":"Processor 1 PUcode Firmware","type":"EXTERNAL"}},{"firmwareVersion":"1.512","systemComponent":{"description":"TrustedPlatformModule20","id":"7bf27b0b-3ff0-4fc7-bfc3-6c255241f817","name":"TPM Firmware","type":"EXTERNAL"}},{"firmwareVersion":"26.37.1700","systemComponent":{"classCode":"2","classSubCode":"0","description":"15b3101f15b30007","did":"101F","id":"d1860c5f-444a-4090-9f15-95d99f249860","name":"Mellanox ConnectX-6 LX OCP3.0","ssid":"0007","svid":"15B3","type":"PCI_DEVICE","vid":"15B3"}},{"firmwareVersion":"20.27.42","systemComponent":{"classCode":"2","classSubCode":"0","description":"14e4165f14e4159c","did":"165F","id":"65855ae1-4015-4c9c-8e02-e2255ac48cd1","name":"BCM 5720 1GbE 2p BASE-T LOM Adptr - NIC","ssid":"159C","svid":"14E4","type":"PCI_DEVICE","vid":"14E4"}},{"firmwareVersion":"26.37.1700","systemComponent":{"classCode":"2","classSubCode":"0","description":"15b3101f15b30012","did":"101F","id":"14d6b86a-01e1-4df4-8e23-7583f0ad86cb","name":"Mellanox Network Adapter - B8:3F:D2:B5:E1:9A","ssid":"0012","svid":"15B3","type":"PCI_DEVICE","vid":"15B3"}},{"firmwareVersion":"1.2.14.1013","systemComponent":{"classCode":"1","classSubCode":"8","description":"1b4b224115900379","did":"2241","id":"677765cf-f642-46ea-aa09-91368297244c","name":"HPE NS204i-u Gen11 Boot Controller","ssid":"0379","svid":"1590","type":"PCI_DEVICE","vid":"1B4B"}},{"firmwareVersion":"2.5","systemComponent":{"classCode":"3","classSubCode":"0","description":"102b0538159000e4","did":"0538","id":"2d22e311-553a-400d-8176-d2b1dda26441","name":"Embedded Video Controller","ssid":"00E4","svid":"1590","type":"PCI_DEVICE","vid":"102B"}},{"firmwareVersion":"HPK4","systemComponent":{"description":"VO003840KYDMV","id":"ca234597-6034-43c0-a909-70b53182ec25","name":"NVMe Drive","type":"EXTERNAL"}},{"firmwareVersion":"HPK4","systemComponent":{"description":"VO003840KYDMV","id":"30b85d69-9764-4c85-8a58-d0a0f2ccf938","name":"NVMe Drive","type":"EXTERNAL"}},{"firmwareVersion":"HPK4","systemComponent":{"description":"VO003840KYDMV","id":"dc9fc833-66f1-467a-b1f3-f99a48b49e1c","name":"NVMe Drive","type":"EXTERNAL"}},{"firmwareVersion":"HPK4","systemComponent":{"description":"VO003840KYDMV","id":"122c9524-1d88-41d5-87cc-da83a095ca17","name":"NVMe Drive","type":"EXTERNAL"}},{"firmwareVersion":"HPK2","systemComponent":{"description":"VR000480KXLXF","id":"20a7ea07-1b32-4fba-92ac-20fe617a0122","name":"480GB 16G NVMe SSD","type":"EXTERNAL"}},{"firmwareVersion":"HPK2","systemComponent":{"description":"VR000480KXLXF","id":"7b3b6d83-77c5-4126-aa96-0535cbff4dba","name":"480GB 16G NVMe SSD","type":"EXTERNAL"}},{"firmwareVersion":"1.04","systemComponent":{"description":"Universal Backplane Manager","id":"b2425f87-8a65-4b3b-9a34-a4ccae84679e","name":"8 SFF 24G x4NVMe/SAS UBM6 BC BP","type":"EXTERNAL"}},{"firmwareVersion":"1.04","systemComponent":{"description":"Universal Backplane Manager","id":"9e6132fe-ee2c-4475-b133-e3133949f0f4","name":"8 SFF 24G x4NVMe/SAS UBM6 BC BP","type":"EXTERNAL"}},{"firmwareVersion":"1.04","systemComponent":{"description":"Universal Backplane Manager","id":"9f199d67-3cdc-4d7c-8628-644554e237bd","name":"8 SFF 24G x4NVMe/SAS UBM6 BC BP","type":"EXTERNAL"}},{"firmwareVersion":"1.04","systemComponent":{"description":"Universal Backplane Manager","id":"9824527b-41c1-4d91-bc6a-082e5a846a16","name":"8 SFF 24G x4NVMe/SAS UBM6 BC BP","type":"EXTERNAL"}}],"hosts":["host-3405617"],"id":"6d669a31-4313-4341-8604-123fab59c1ff","messages":["Error occured during inventory of host-3405617"],"operationStatusCode":500,"progress":100,"startTime":"2024-05-14 13:52:12.612105","status":"FAILED"}

[ndavis_cmm]

Good to know that works! We have a case with HPE, if we find a better solution I will share it! Thank you! 

[ndavis_cmm]

Not yet, still working with HPE. Apparently reverting OneView should work, but I would prefer not to do that. I will update the post if I find anything! 

[ndavis_cmm]

Hello,

HPE Support Case Number: 5381923534

Thank you! 

[ndavis_cmm]

Update:

The only fix we have found so far is to re-apply the Server Profile to the compute module and then it will allow it to perform the compliance check on the sytem. This is not a great solution. We are pushing HPE to find a better solution. 

For our upgrades of OneView from 8.5 > 8.6 > 8.7 etc we did not have to reapply the Server Profile. 

[pirx]

I do not even have profiles as I only use monitored mode. That's all I need together with vLCM (in case this ever starts working to be stable)

[rkiik]

Interestingly for me even the Server Profile re-apply did not solve the issue.
But even when it would work, it's quite silly to think that you would re-apply profiles for 20-50-100 machines.

[ndavis_cmm]

I completely agree we have over 100 hosts in this environment. We are waiting on more information from support.

I will say we noticed some oddities when doing the compliance check at the cluster level vs the individual server. If we tried the cluster the compliance check would fail still, but if we selected one of the hosts we re-applied the server profile it would work. 

Hopefully, they will find a solution quickly. 

[Rama2]

I see that HPE OneView L3 engineers are working on issue - You should receive update's via HPE case 5381923534.

[pirx]

Any update on this? From the experience with my OV cases, the probably is high that - by accident - a new version will fix an issue.

[ndavis_cmm]

We get a weekly email that they are working on it  

Nothing substantial as of yet

[pirx]

@ndavis_cmm sorry to bother you but was there any progress?

[ndavis_cmm]

No worries! We have not gotten to a resolution thus far.

This was our last update: 

Following is the update from HPE engineering team,

We understand that configuration uses ‘High Security’ security state in iLO settings. While this configuration has been functional up to some point, it is important to note that OV4VC officially supports only the 'Production' security state as mentioned in the OV4VC User Guide.

Any other security state, including 'High Security', is not qualified by HPE for use with OV4VC.
Using an unsupported security state, such as 'High Security', may result in operational anomalies or inaccuracies that could impact on your system's reliability. To ensure the proper functionality of OV4VC and maintain the integrity of your system, we strongly recommend configuring your iLO settings to the 'Production' security state.

The funny thing, we literally tried this on the call with them and it did not resolve the issue. We have pushed back on that non-solution and are awaiting a response, but I am not optimistic at this point. Reapplying the server profile with high security still enabled is the only thing that seems to have worked for us. It might be worth tryin to disable high security in your environement if you have it enable the iLO, but that did not work for us. 

Thank you,

Nate

[robinharmsen]

We have been updating our OneView into version 8.90 yesterday and now I am facting exactly this issue.

Hope a viable solution will soon be available

[ndavis_cmm]

Have you put in a ticket? We are getting a bit of a run-around from support. I'm hoping we can get everyone with this issue to poke support until we actually get some movement on this issue. 

Thanks! 

[robinharmsen]

I did not yet create a ticket for this.

Thought this might had to do with another issue we had with our environment  the famous "Unable to get type data" ... had returned.
Had that fixed with support this morning.. to notice we still have issues with vLCM.

I did some testing and for us these steps seem to work.

- Do a profile reapply for the corresponding server
- Do a Pre-Check in vLCM for the server
- Do a remidiate for the server

As we have large clusters this is not really feasable, but it might give some indication as to where the issue is in OneView.
Also we do not seem to have issues with clusters in our other (older) OneView instances.. so this is realy an issue whith this OneView version.

Will try to create a ticket with support tomorrow to raise the issue count at HPE to get this fixed sooner hopefully 

[robinharmsen]

I created a case (5382909305) hope it helps in getting this solved soon by engineering

[robinharmsen]

We are now also getting advice to set the iLO settings to 'Production' security state

Thing is... all our servers are already on Security State Production