catch 22, Unable to discover adapter information. Locked

[pirx]

I've added a server to OneView. This task is still somewhat stuck with "Unable to discover adapter information.". I've already powered off/on the server but I only get "Unable to refresh the server because the server hardware is currently being added to the appliance." when I try to refresh.

So what can I do now? I don't see any option to cancel the "add" task. When will this finally time out, it now running 2 hours.

Unable to discover adapter information.

8/3/23  2:01:01 pm

2 hours ago

Locked

unassigned

Resolution 

Power cycle the server, allow it to complete Power On Self-test and then refresh the server hardware.

Unable to discover adapter information (hpe.com)

[ChrisLynch]

This typically happens when one of a few things goes wrong:

1. Older ILO5 firmware causes a device discovery subtask from completing successfully. This was addressed around the 2.5x ILO5 firmware releases.
2. OneView 7.x had a workaround for the above iLO issue but didn't fix all cases. OneView 8.0 should be used at a minimum instead.

Rebooting or powering off the server won't address this as iLO is used to discover hardware, not the OS installed.

[pirx]

ILO is version 2.91 and OV latest 8.4.  The task has the same state as 17h ago, "Unable to discover adapter information. Locked"

What other options do I have? I noticed in my first tests months ago that certain task can not be canceled, which was an imidiate no go for continuing the POC.   For example firmware update tasks.. There were also some "unexpected issues" and the task retried over and over. No chance to cancel from withing OV. 

I've now also reset ILO, still no change. I just can't refresh the server as the "add" task is still active. 

[pirx]

@ChrisLynch 

Ok, this is just broken. Yesterday when the error occured first, I added 4 hosts in the same task as "Managed" and the first host in the list is still stuck. Then I added some more, I think everytime just once in a task., no errors. Now I added again 4 host in one task an again the first one in the list is stuck "Unable to discover adapter information. Locked". OV is latest version, all ILOs are on 2.91. I have no debug logs but this smells like a bug in OV.

[ChrisLynch]

Do you have a firewall between the appliance and iLO's?  We have not had any customers report this type of issue.  If you haven't opened support case, I would ask that you do so an appliance support dump can be then analyzed.

[pirx]

There is a firewall, but this does not expain why one out of four ILOs is failing. Or why OV is able to continue with discovery once the applinace is rebooted, that is how I solved it for now.

But I understand that there is no documented way to break this lock other than reboot, right?

[ChrisLynch]

At this point, I would ask that you open a support case here.  We have had situations where customer firewalls have caused intermitent issues when placed between the appliance and iLO.  OneView performs different operations to iLO than iLO Amp would.  

[DonZoomik]

I also have this problem. One thing that I've noticed - when you disable NIC Option ROM, discovery seems to always fail for at least some adapter types (534 mostly I think).  They were disabled a long time ago when VMware qfle3f drivers had crashing problems with unused FCoE. After re-enabling Option ROM, discovery mostly works.

However also in my case, after server boot adapters are discovered but about a week later the same alarms pop up  (Unable to discover adapter information). They will only clear after server reboot only to return some time later. No firewalls here. OV 8.40, iLO5 2.91.

It's also a trial/demo setup so I don't think support will help here...

[ChrisLynch]

Can you please supply the exact adapter model and partnumber you have installed, along with the installed firmware version?  We have seen this as adapters move to a Redfish model for inventory and management.  Needless to say, we have run into bugs and issues with implementations.  Both iLO and OneView have implemented some workarounds for certain cases.

[DonZoomik]

In this environment, there are mainly DL360G10 pre-NC models (with onboard 1G ports) that mainly show issues. Onboard 1G ports are fully disabled as they are not used at all - I expect that these are not the problem as they don't show up at all anywhere after disabling, FlexLOM and PCIE standup cards do show up normally, despite the error.

Remaining NICs from OV:

Embedded ALOM HP FlexFabric 10Gb 2port 534FLR-SFP+ Adapter
Firmware: 7.19.14
Part number 701531-001
Assembly number 700749-001
Serial number **Confidential info erased**

PCI-E Slot 2 Mellanox Network Adapter - B8:3F:D2:59:1F:78
Firmware 26.35.1012
Part number
Assembly number P42046-001
Serial number **Confidential info erased**

Firmware versions sometimes don't match between what support site shows but firmwares were updated from G10 SPP 2023.03.02.

[DonZoomik]

Second system type currently giving errors:

Apollo 4200 G10 ("ProLiant XL420 Gen10"). Same case, onboard 1G NICs are disabled in BIOS. Remaining cards:

PCI-E Slot 1 HP Ethernet 10Gb 2-port 530SFP+ Adapter
Firmware 7.19.14
Part number 656244-001
Assembly number 652501-001
Serial number **Confidential info erased**

PCI-E Slot 2 Mellanox Network Adapter - B8:3F:D2:59:1F:4C
Firmware 26.35.1012
Part number
Assembly number P42046-001
Serial number **Confi7dential info erased**

[pirx]

@ChrisLynch 

Two of my servers have again switched to "Unable to discover adapter information. Locked" status. No fw update recently, port map was discovered fine before. Server uptime is 21 days.

[ChrisLynch]

Please work this issue with HPE Services and a support case.  You are using the most current firmware versions posted, and I can't find any known issues related to these versions.  Please be prepared to provide an appliance support dump for analysis.

[ChrisLynch]

There is a Customer Advisory on this issue, if specific Broadcom adapters are being used when updating to 226.1.107.0 (bundled in Service Pack for ProLiant 2023.09 Gen10).  This will require an iLO5 and 6 firmware update that we will be releasing soon.