topic Re: ESXi host memory health warning - iLo 5 v2.30 in Operating System - VMware

ESXi host memory health warning - iLo 5 v2.30

PatrickLong — Fri, 11 Sep 2020 12:57:50 GMT

There appears to be an issue with new Gen10 iLo5 firmware 2.30 and ESXi 6.7 memory health warnings. After upgrading my Synergy SY480 and SY660 blades to the latest 2.30 release of iLo5 firmware, within a short while (not immediately) all of these hosts are showing a red bang error for Host Memory Status. If I Reset to Green, the error clears for a short while, but then eventually returns.. On important item I've noticed is that when the host is in the Error state under iLo5 2.30 , under Monitor > Hardware Health I see new sensors for memory with names like "Memory Device 1 Mem_Stat_C01S02" in Alert status with Reading of either 128 or 192 as shown:

However, when I back-rev the iLo to previous version 2.18 and reboot the host, the errors are gone as are the newer memory sensor readings:

So it appears that iLo5 2.30 introduced NEW sensor readings for memory that ESXi is not handling properly - and I am using the very latest build release VMware ESXi 6.7.0 build-16713306 release date 8-20-2020 and all latest firmware/drivers/HPE utils esxi6.7uX-mgmt-bundle-3.5.0-12 and esxi6.7-util-bundle-3.5.5-3, as well as latest BIOS I42 v2.36 7/16/2020, Innovation Engine 0.2.2.0, and Server Platform Services 4.1.4.381.1. As well as latest OneView Composer 5.40.00-0423953\. This seems very similar to this old sensor-reading chestnut from 2018/19: with ESXi and iLo5 v1.30 https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00053955en_us

Anyone else experiencing the same issue? I don't have any non-Synergy Gen10 hosts to test this on to see if the issue is consistent across all Gen10 / iLo5 systems and ESXi but my guess would be that it is. And yes, I'm well aware that the latest Synergy Custom SPP 2020.07.01 (or " Service Pack for ProLiant, SY-2020.07.1" as it's apparently being called now - but don't try running it on a ProLiant server, it won't work) includes iLo5 v2.18, NOT 2.30 - however the 2.30 fixlist specifically includes a fix for Synergy:

"Synergy compute modules do not power up after One-button secure erase completes."

which makes it a bit of a hard sell to say "Don't install 2.30 on Synergy".

Re: ESXi host memory health warning - iLo 5 v2.30

netscaper71 — Sat, 12 Sep 2020 18:48:44 GMT

Hello, we have same issue exactly same. 6.7 fresh updates from friday and ilo5 ver2.30 on 5 servers
ProLiant DL360 Gen10
Memory Device 3 Mem_Stat_C01S04

Re: ESXi host memory health warning - iLo 5 v2.30

ArminasU — Mon, 14 Sep 2020 08:37:25 GMT

Hi, I have same issue on all hosts ProLiant DL360 Gen10. Updated ESXi to version 6.7 16713306 and ILO5 to v2.30.

ESXi hardwre health Alerts:

Memory Device 10 Mem_Stat_C01S11

Memory Device 11 Mem_Stat_C01S12

Memory Device 18 Mem_Stat_C02S03

When we can expect new ILO5 update version with fix or i have to downgrade to ILO5 version 2.18?

Re: ESXi host memory health warning - iLo 5 v2.30

Mountain — Mon, 14 Sep 2020 15:04:47 GMT

Same to us, DL380Gen10 - put it back to 2.19 Was a advise from support and they are in contact with vmware

Re: ESXi host memory health warning - iLo 5 v2.30

David Kobler — Mon, 14 Sep 2020 15:21:37 GMT

same here

Re: ESXi host memory health warning - iLo 5 v2.30

fojtp — Tue, 15 Sep 2020 09:13:44 GMT

Gen10 hosts affected (BL460c, DL380) with ESXi versions (6.5 - tested latest build 16576891, 6.7 - build 16316930).
ESXi 7.0 (latest build 16324942) seems OK because Sensor Status is Uknown (for Memory Device)

Re: ESXi host memory health warning - iLo 5 v2.30

MK_J — Wed, 16 Sep 2020 05:24:52 GMT

It seems iLo5 FW 2.30 exposes some additional sensors and alerts are triggered due to this.

To avoid getting these memory alerts, you may remain at N-1 FW version.

Please open a support case with HPE Synergy so that you would be updated with develpoment on this issue.

Best Regards,

Milan

Re: ESXi host memory health warning - iLo 5 v2.30

AmritRanjan — Wed, 16 Sep 2020 07:37:32 GMT

This issue is already under investigation at HPE.

If you feel this was helpful please click the KUDOS! thumb below!

Regards,

Re: ESXi host memory health warning - iLo 5 v2.30

SanjeevGoyal — Wed, 16 Sep 2020 07:36:41 GMT

Hello,

I hope all is well.

I would suggest you to have a proper case be logged with HPE, and share the appropriate logs for further analysis.

If you feel this was helpful please click the KUDOS! thumb below!

Regards,

Re: ESXi host memory health warning - iLo 5 v2.30

David Kobler — Wed, 16 Sep 2020 16:47:22 GMT

Is there an oficial HPE document/advisory for the issue?

Re: ESXi host memory health warning - iLo 5 v2.30

MK_J — Thu, 17 Sep 2020 01:46:13 GMT

This issue is currently under investigation at HPE but no Advisory has yet been released.

Re: ESXi host memory health warning - iLo 5 v2.30

AmritRanjan — Thu, 17 Sep 2020 14:24:18 GMT

Yeah! I witnessed this too.

Could you please log a support case and send me a private message with the case number? There is a requirement also to log a case with VMware as it is not able to read IPMI information correctly on iLO firmware v2.30.

Something similar to https://kb.vmware.com/s/article/53134?lang=en_US

Re: ESXi host memory health warning - iLo 5 v2.30

RajSanthanakris — Sat, 19 Sep 2020 17:19:00 GMT

A case has been logged with vmware and we are working closely with them. We will escalate it as necassary.

Re: ESXi host memory health warning - iLo 5 v2.30

Stuart Green — Mon, 21 Sep 2020 14:43:17 GMT

Can you downgrade iLO firmware from the iLO web management page? Just by uploading the ilo5_219.bin file ?

I got an error back saying firmware upgrade in progress....hmm.

Thanks for insight and advice.

I have a case open with VMware and HPE on this also.

ESXi 6.7 and DL380 Gen10 on iLO 2.30...

I have a System Chassis Unknown status also
See Post 11. here
https://communities.vmware.com/message/2984294

Re: ESXi host memory health warning - iLo 5 v2.30

AmritRanjan — Mon, 21 Sep 2020 16:22:30 GMT

Stuart,

Thanks for helping with updates.

Please drop me a personal message with HPE case number and VMware SR.

I have seen chassis displaying unknown on iLO3/4/5 and IPMI data from iLO is always reported fine.

It is OS interpretation to display the correct status in sensors tab.

Please check the status from iLO accessing IPMI data. IPMI is disabled and requires to turn from iLO GUI.

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sensor list

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sdr list all

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sdr elist all

https://support.hpe.com/hpesc/public/docDisplay?docId=c04530505

Re: ESXi host memory health warning - iLo 5 v2.30

ArminasU — Tue, 22 Sep 2020 07:09:26 GMT

We downgraded with fille ilo5_219.bin without esxi reboot. Error disappeared.

Re: ESXi host memory health warning - iLo 5 v2.30

RajSanthanakris — Tue, 22 Sep 2020 07:21:38 GMT

Please note it is a workaround. Because we enabled memory sensor records in iLO 5 v2.30, the issue still need to get root caused and fixed in ESXi. We are working closely with vmware on this.

Thanks,
Rajagopal

Re: ESXi host memory health warning - iLo 5 v2.30

SweMatte — Tue, 22 Sep 2020 12:41:10 GMT

Oops never mind....

Re: ESXi host memory health warning - iLo 5 v2.30

UlrichThomas — Sat, 26 Sep 2020 08:42:42 GMT

Any news on this topic?

Re: ESXi host memory health warning - iLo 5 v2.30

Nikolay Savov — Mon, 28 Sep 2020 13:20:26 GMT

Hello colleagues from the forum.

I hit the same problem also 1:1 !

The good news is that I found a workaround as some had advised.

Configuration:

ProLiant DL360 Gen10 ilo 2.30

VMware-ESXi-6.7.0-Update3-15160138 - HPE-Gen9plus-670.U3.10.5.0.48-Dec2019.iso

Workaround:

1. UPDATE server FW with HPE SPP - P26941_001_gen10spp-2020.03.2-SPP2020032.2020_0831.6.iso that will give you option to downgrade to iLO 2.14

2. Full Vmware update - patch + fix to ver. VMware ESXi, 6.7.0, 16713306

And 5 days already no RED ALERTS ....

Goot luck with the fixing this issue guys from HPE and Vmware!

Kind Regards