ESXi host memory health warning - iLo 5 v2.30

[PatrickLong]

There appears to be an issue with new  Gen10 iLo5 firmware 2.30 and ESXi 6.7 memory health warnings.  After upgrading my Synergy SY480 and SY660 blades to the latest 2.30 release of iLo5 firmware, within a short while (not immediately) all of these hosts are showing a red bang error for Host Memory Status.  If I Reset to Green, the error clears for a short while, but then eventually returns..  On important item I've noticed is that when the host is in the Error state under iLo5 2.30 , under Monitor > Hardware Health I see new sensors for memory with names like "Memory Device 1 Mem_Stat_C01S02" in  Alert status with Reading of either 128 or 192 as shown:

However, when I back-rev the iLo to previous version 2.18 and reboot the host, the errors are gone as are the newer memory sensor readings:

So it appears that iLo5 2.30 introduced NEW sensor readings for memory that ESXi is not handling properly - and I am using the very latest build release VMware ESXi 6.7.0 build-16713306 release date 8-20-2020 and all latest firmware/drivers/HPE utils esxi6.7uX-mgmt-bundle-3.5.0-12  and esxi6.7-util-bundle-3.5.5-3, as well as latest BIOS I42 v2.36 7/16/2020, Innovation Engine 0.2.2.0, and Server Platform Services 4.1.4.381.1. As well as latest OneView Composer 5.40.00-0423953\.  This seems very similar to this old sensor-reading chestnut from 2018/19: with ESXi and iLo5 v1.30  https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00053955en_us

Anyone else experiencing the same issue?  I don't have any non-Synergy Gen10 hosts to test this on to see if the issue is consistent across all Gen10 / iLo5 systems and ESXi but my guess would be that it is.  And yes, I'm well aware that the latest Synergy Custom SPP 2020.07.01 (or " Service Pack for ProLiant, SY-2020.07.1" as it's apparently being called now - but don't try running it on a ProLiant server, it won't work) includes iLo5 v2.18, NOT 2.30 - however the 2.30 fixlist specifically includes a fix for Synergy:

"Synergy compute modules do not power up after One-button secure erase completes."

which makes it a bit of a hard sell to say "Don't install 2.30 on Synergy".

[netscaper71]

Hello, we have same issue exactly same. 6.7 fresh updates from friday and ilo5 ver2.30 on 5 servers
ProLiant DL360 Gen10
Memory Device 3 Mem_Stat_C01S04

[ArminasU]

Hi, I have same issue on all hosts ProLiant DL360 Gen10. Updated ESXi to version 6.7 16713306 and ILO5 to v2.30.

ESXi hardwre health Alerts:

Memory Device 10 Mem_Stat_C01S11

Memory Device 11 Mem_Stat_C01S12

Memory Device 18 Mem_Stat_C02S03

When we can expect new ILO5 update version with fix or i have to downgrade to ILO5 version 2.18?

[Mountain]

Same to us, DL380Gen10 - put it back to 2.19   Was a advise from support and they are in contact with vmware

[David Kobler]

same here

[fojtp]

Gen10 hosts affected (BL460c, DL380) with ESXi versions (6.5 - tested latest build 16576891, 6.7 - build 16316930).
ESXi 7.0 (latest build 16324942) seems OK because Sensor Status is Uknown (for Memory Device)

[MK_J]

It seems iLo5 FW 2.30 exposes some additional sensors and alerts are triggered due to this. 

To avoid getting these memory alerts, you may remain at N-1 FW version.

Please open a support case with HPE Synergy so that you would be updated with develpoment on this issue.

Best Regards,

Milan

[AmritRanjan]

This issue is already under investigation at HPE.

If you feel this was helpful please click the KUDOS! thumb below!   

Regards,

[SanjeevGoyal]

Hello,

I hope all is well.

I would suggest you to have a proper case be logged with HPE, and share the appropriate logs for further analysis.

If you feel this was helpful please click the KUDOS! thumb below!   

Regards,

[David Kobler]

Is there an oficial HPE document/advisory for the issue?

[MK_J]

This issue is currently under investigation at HPE but no Advisory has yet been released.

[AmritRanjan]

Yeah! I witnessed this too.

Could you please log a support case and send me a private message with the case number? There is a requirement also to log a case with VMware as it is not able to read IPMI information correctly on iLO firmware v2.30.

Something similar to https://kb.vmware.com/s/article/53134?lang=en_US

[RajSanthanakris]

A case has been logged with vmware and we are working closely with them. We will escalate it as necassary. 

[Stuart Green]

Can you downgrade iLO firmware from the iLO web management page? Just by uploading the ilo5_219.bin file ?

I got an error back saying firmware upgrade in progress....hmm.

Thanks for insight and advice.

I have a case open with VMware and HPE on this also.

ESXi 6.7 and DL380 Gen10 on iLO 2.30...

I have a System Chassis Unknown status also
See Post 11. here
https://communities.vmware.com/message/2984294

[AmritRanjan]

Stuart,

Thanks for helping with updates.

Please drop me a personal message with HPE case number and VMware SR.

I have seen chassis displaying unknown on iLO3/4/5 and IPMI data from iLO is always reported fine.

It is OS interpretation to display the correct status in sensors tab.

Please check the status from iLO accessing IPMI data. IPMI is disabled and requires to turn from iLO GUI.

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sensor list

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sdr list all

ipmitool -I lanplus -H <iLO_IP> -U <User_Name> -P <password> sdr elist all

https://support.hpe.com/hpesc/public/docDisplay?docId=c04530505

[ArminasU]

We downgraded with fille ilo5_219.bin without esxi reboot. Error disappeared.

[RajSanthanakris]

Please note it is a workaround. Because we enabled memory sensor records in iLO 5 v2.30, the issue still need to get root caused and fixed in ESXi. We are working closely with vmware on this.

Thanks,
Rajagopal

[SweMatte]

Oops never mind....

[UlrichThomas]

Any news on this topic?

[Nikolay Savov]

Hello colleagues from the forum.

I hit the same problem also 1:1 !

The good news is that I found a workaround as some had advised.

Configuration:

ProLiant DL360 Gen10   ilo 2.30

VMware-ESXi-6.7.0-Update3-15160138 -  HPE-Gen9plus-670.U3.10.5.0.48-Dec2019.iso

Workaround:

1. UPDATE server FW with HPE SPP - P26941_001_gen10spp-2020.03.2-SPP2020032.2020_0831.6.iso that will give you option to downgrade to iLO 2.14

2. Full Vmware update - patch + fix to ver. VMware ESXi, 6.7.0, 16713306

And 5 days already no RED ALERTS ....

Goot luck with the fixing this issue guys from HPE and Vmware!

Kind  Regards

[AmritRanjan]

Customer Advisory is published on this issue.

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00106468en_us

[Sysadminm]

We have the same issue.  I just updated all our hosts to ILO 5 v2.30 and within a short time vCenter received the same errors.  It is not specific to the synergy blades.  We have a few stand alone DL380s which also get this error after the ILO update.

[Sysadminm]

Also note we are still running ESXi 6.0 as well.

[AmritRanjan]

All iLO5 based servers will show these additional sensors on firmware v2.30.

OS: - ESXi6.0/6.5/6.7/7.0

[UlrichThomas]

Hello, I wonder why such updates are not properly tested before release. Most customers will use VMware and encounter this issue.

Does HPE not test in a VMware environment?

This should definitely be improved in the future, or with new releases also provide an appropriate VMware update.