Oneview 6.3 and ILO5 2.55 issue

[PriitP]

Hello, community!

Last week HPE released a new SPP Gen10 2021.10.0 which also includes ILO5 firmware 2.55. I tried it in our test setup. Uploaded it to OV and updated server profile (DL360Gen10 and ESXi 7.0U3) Everything worked as expected and server got updated. Server profile shows green dot, that it's compliant. Meanwhile server hardware object turned yellow showing like something is wrong with actual hardware. Warning message states: "Unable to fully discover the port map information for the server hardware."  In ILO5 all adapters and components are properly discovered and no issues detected. OV is also listing all components and their firmwaress correctly. Server hardware refresh ends also with error referring to the same warning. Needless to say, that with firmware 2.44 there were no issues.

Rgds

Priit

[ivarh]

Hi,  I don't have an answer, but see kind of similar. Upgraded a Gen9 server yesterday w SPP 2021.10, and as a test I portdisabled a SAN port to this server. In ILO (now v 2.78) I have a yellow degraded state on the fiber port, but OneView (v 6.30.00) shows server as green. ILO status is not propagated to Oneview..

[PriitP]

Yes, your problem is a bit different. Today i added an new DL380 Gen10 server as monitored to the same Oneview appliance, also with latest ILO5 firmware 2.55. Since monitored mode does not display any port information, the error message also didn't pop up. I don't have any Gen9 servers in managed mode in Oneview, but all monitored mode servers are with ILO4 firmware 2.78 and are working properly. Also real link down issues are detected and resolved as expected.

[derick_souza]

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00113227en_us

[ivarh]

Hi guys, Thanks for replying  

Just to make sure what I experience:

I have done this action a lot of times to show others. This is the first time I see the Degraded status is not propagated to OV.  This is the only server yet upgraded w SPP 2021.10.

[AmRa]

With reference to Degraded status is not propagated to OV, Check by reset iLO from iLO web console and later refresh the server from OneView Console.

Procedure to Rebooting (Resetting) the iLO processor with the web interface

1. Click Information in the navigation tree, and then click the Diagnostics tab.
2. Click Reset..
3. When prompted to confirm the request, click OK.
4. iLO resets and closes the user's browser connection.
5. Wait for sometime (30-40 sec) and later check the status.

[PriitP]

Although the warning is the same, scope is different. I don't have Gen10+ servers nor referred Intel NICs. My servers have Mellanox NICs and Qlogic fc cards. Like mentioned earlier, issue arises right after ILO firmware upgrade. It also doesnät even matter, if i update other firmwares.

[PriitP]

I just did a downgrade of ILO5 firmware to 2.44 and the warning disappeared. Now, of course, the profile is out of compliance. So, this is purely 2.55 firmware and OV combo issue.

[ivarh]

Hi folks, thanks for replying.  I have reset the ILO (earlier) - no change.

I have removed and "Added new hardware" in OV - no change

I have some other servers that I've only upgraded ILO4 (to 2.78).
I did another test against one of them, and it did not propagate to OV either.
(Haven't really tested earlier.)

It seems like the newest ILO FW (both ILO5, in your case PriitP, and ILO4 in mine). has an issue.

I'll try to log a case somwhere, because right now we cannot rely on Oneview..

I'll be back

[PriitP]

Interesting finding: then using a DL360Gen10 server with HPE Ethernet 10Gb 2-port 562FLR-SFP+ adapter, no warning after updating hardware with latest SPP and ILO firmware 2.55.. Also remediating host with VMware vLCM image, that includes latest HPE SPP2021.10.0 everything goes smoothly. But DL360Gen10 with HPE Eth 10/25Gb 2p 640SFP28 Adptr not only getting Oneview warning, but also fails remediating a host. Oneview activity shows, that it flashes ILO and UEFI and then stops, until getting 120 minutes timeout. After manually rebooting the server and reapplying the profile in Oneview it continues and finishes the updating. So looks like OV 6.3 in combination with ILO5 fw 2.55 does not like HPE Eth 10/25Gb 2p 640SFP28 Adapters.

[ivarh]

Hi PriitP,

I have logged a case and have a dialogue w Onview Support. I've made a kind of matrix on what works and
what doesn't work regarding FC-links (Other part may work even though FC doesn't..), ILO fw, Server types, licenses ++, on some of my servers, and provided to support.
In logging this case I've pointed them to this post, and they know about it. (So they'll probably notice your well-thought comments as well :))

I used to rely next to 100% (!?) on the product. I think it does an excellent job, and I beleive we'll get the confidence back soon
Hopefully..

Regards
Ivar   :^)

[Felix Taskar]

I noticed this in the relase notes for iLO5 v2.55: Dependance on AMS new version 2.5.0, are you running that in the OS?

https://support.hpe.com/hpesc/public/swd/detail?swItemId=MTX_f9e090d9fc9f45bb849716b22c#tab-history 

FIRMWARE DEPENDENCY:

  Please note the supported version of AMS with iLO 5 v2.55 is 2.5.0 for Windows and Linux components, while it is 2021.10.01 for VMware component.

  Failure to use this version will result in loss of manageability of storage components that are managed by AMS. The behavior of iLO will be unpredictable if this version of AMS is not used with iLO 5 v2.55.

[PriitP]

AMS driver is up2date. 701.11.8.0 In fact ESXi image is built with latest HPE Add-on bundle.

https://support.hpe.com/hpesc/public/swd/detail?swItemId=MTX_ef3c7b0fd13e4ee486e0263676#tab-history

[ivarh]

Hi,

I saw this support alert was released today:

Document - Advisory: HPE Integrated Lights-Out 4 (iLO 4) and iLO 5 - Remote Support Alerts May Not Be Sent by iLO 4 or iLO 5 | HPE Support

I still have an ongoing case, and engineering has "replicated the issue". This Advisory might be a result from that..

Happy testing

-Ivar

[PriitP]

I also opened a support case on my problem. So far HPE asked me to reset all servers to factory default, then reboot ILO and start server up again. Then reapply Oneview server profile. Even though they claimed, it should permanently fix the issue, it didn't. In between, there has bee released a new ILO 5 firmware 2.60, but that one does not fix the issue. So i'm still waiting for response from support.

[MikeM3]

Confirming the same here.  Have tried everything in the list with the issue, no luck at getting it to clear so far.

Gen10 DL380, iLO v2.55 and OneView v6.40.  Occuring identically on multiple machines. Screenshot attached.

[Bob Curnock]

Downgrade ilo firmware to 2.48 and problem goes away -