ESXi host memory health warning - iLo 5 v2.30

[JoeAcosta]

Exactly my thoughts.  This issue occured on all of my BL460c Gen10s, running the current recipe. 

I thought recipes were tested by HPe.  How does something so blatantly obvious get missed?   Seriously - these issues appeared in minutes to hours in my environment, with no "special circumstances" to make the problem occur.

[Keysteal]

Same here. New Hp Proliant with Esxi 6.7U3 and latest ILO installed. So the solution for now is to ignore the errors, right?

[RajSanthanakris]

Yes, we don't see any functional impact. This issue is being actively worked upon by VMware engineering team.

Thanks,
Rajagopal

[EPuser]

We raised this with HPE as soon as it was released and unfortunately they were not helpful.  They simply closed the case and gave me some commands to run to check the memory status via SSH.  If you use HpeIloCmdlets via PowerShell to manage iLO you will notcie this is also broken as well as VMware host health reporting.  When trying to run simple commands such as to return firmware versions you get no data. Again the repsonse from HPE? Sent me some commands to check memory status. I asked about how the updates were tested and how we got in this mess but no response.

[Keysteal]

it depends on the person supporting you, this is not an answer, because I think all the people here can check the memory status in ILO, without using the commands in the terminal.

[Hongva]

Same issue with the BIOS 2.22 version, I suggest you to revert back to 2.10. No issue encountered with hpe servers running 2.10. good luck with you

[Paulw1984]

Same for me on

VMware ESXi, 7.0.0, 16324942

[answer-is-42]

Same Problem here with ESXi 6.7 (recent Build) on several DL385 Gen10 (incl. Plus Modells) with all the FW updates available right now ...

As the Advisory is mentioning, this is a current Problem with ILO5 2.30 on all Gen10 Systems, was hoping that for some reason the AMD-based ones where lucky...

Please come up with a solution for this issue, cause a lot of customers are asking uncomfortable questions ...

[RajSanthanakris]

Yes, we are making a short term fix in iLO 5 v2.31 as we work through the final solution with VMware. Will update once we get iLO 5 v2.31 released.

Thanks,
Rajagopal

[emaneiro]

Hi,

We have a customer with this errors after an iLO firmware upgrade. Apart from errors in vCenter, has this caused server crashes or any problem affecting server availability?

Regards

[AmritRanjan]

No disruption reported on Server availability.

[wreigle]

Is there an ETA available for iLO 2.31 that contains a fix?

[RajSanthanakris · Accepted solution]

It has been signed off, should be available for download today or tomorrow.

Thanks,
Rajagopal

[PatrickLong]

As @RajSanthanakris mentioned above, the iLo5 v2.31 is now available for download.  It seems to have resolved the memory health warning issue on the single DL385 Gen10 Plus host on which I installed it.  I will not be able to roll the new version out to additional Gen10 hosts tonight, but this is a promising start.

[AmritRanjan]

iLO5 Firmware v2.31 is available on https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_719b3c76e0eb47ab98073490d9#tab5

A permannet fix is been worked within VMware - HPE .

[AmritRanjan]

A quick note on Synergy.

iLO Firmware v2.30 is not certified on any Synergy Nodes – iLO5.

For supported releases on Synergy please visit https://techhub.hpe.com/us/en/enterprise/docs/index.aspx?doc=/eginfolib/synergy/sw_release_info/index.html

[PatrickLong]

Thank you, @AmritRanjan - I am aware that for iLo5, neither v2.30 released Sept 4, 2020 , nor newer v2.31 released Octover 20, 2020, nor older v2.19 released Aug 18, 2020 are supported on Synergy.  The latest iLo5 release officially supported on Synergy is still v2.18 released July 10, 2020 and included as part of the Synergy_Custom_SPP_2020.07.02_Z7550-97031.iso.

I have had this discussion before here in the HPE Communities and I continue to follow the guidance given to me then: "It is never supported to download components outside the HPE Synergy Custom SPP (except for elevations from our engineering response team to serious issues)".  I do appreciate the additional complexity of validation in the Synergy ecosystem - I just wish the certification process for this Enterprise-class solution did not lag so far behind other HPE product lines.  As a customer it is frustrating to see fixes pertaining to Synergy in the "Fixes" tab for these new iLo versions that I'm not supposed to use.  I hope you can appreciate that it is at best a mixed message to list a Synergy-related fix (like this one for v2.30) "Synergy compute modules do not power up after One-button secure erase completes." and then also say "but don't load this version on Synergy iLo's - it's not supported"

I continue to hope that one day iLo updates for Synergy will be decoupled from strict adherance to the current Synergy SPP - in the same way that Synergy BIOS updates were decoupled some time ago,

[EliteX2Owner]

I just deployed a big batch of DL385 g10's and of course ran into this issue, updated to 2.31, and all the memory errors cleared, but the servers have since gone back to alarm state due to:

System Chassis 1 UID  - Unknown

Resetting to green seems to last a day or so before reverting.  This is in 6.7 build 16773714.

Honestly I don't blame HP for these issues; vmware shouldn't stupidly assume that seeing new hardware sensors is grounds for considering them to be in an alarm state, and other vendors may want to see the new sensors.  It would be nice however if HP let you turn sensor visibility off selectively in iLo; solves the problem for vendors like vmware who are playing catch up.

[David Kobler]

Has anyone tried on a dl380 gen 10 running esxi 6.5 ? It's unfortunate customers have to rely on other customers for basic compatibility testing. It seems the certified designation doesn't hold much, as basic testing is not done by the vendor.

[Keysteal]

I'm reading that the latest firmware fixes the memory bug, but we may have more. So, is it safe or not to apply the new ILO firmware?

[FrostByteVA1]

For what it's worth, we have a Gen9 w/ iLO5 with the same memory errors after upgrading from 2.14 to 2.3.  I've updated to 2.31 and reset the alarms.  Currently monitoring the results.

[Mattai]

Five servers from 2.31 are working fine.
IPMI read the data correctly

[ICTWBW-MER_beh]

Any indication on when this permanent fix will be released ? This month ? Next year ?

[Mattai]

For DL385 G10 - is available 2.31 Oct 13 2020

[FrostByteVA1]

2.31 is out.  It has taking care of our memory errors but I'm still seeing the following error, in SolarWinds SAM, even though the iLO shows healthy.

SENSOR WITH PROBLEMS: CPU (Critical): Processor 1 CPU_Stat_C1: IERR (Critical)