*PSOD* with iLo6 v1.60

[Marcel_D]

Hi,

today we've encountered a heavy issue with following combination:

DL360 Gen11

iLo6 v1.60

vSphere 8 U2

When we manually trigger an iLo Reset, we get a PSOD on the Server, while running. This is reproduceable anytime with iLo Version v1.60.

When we downgrade back to 1.57, the PSOD just happens on the first reset, because it is still running on 1.60. After it comes up with 1.57 and trigger a reset, the PSOD-Issue is gone.....

I've found a similar Issue with Gen 10 Server in this Forum: https://community.hpe.com/t5/proliant-servers-ml-dl-sl/psod-with-vsphere-8/td-p/7191848/page/2

I d not know, if HPE is aware of this or this is just releted to us , but IMHO this is a highly critical issue. I haven't opened a case yet.

Regards

[shiva_jr]

Hi @Marcel_D,
    Please find the step, which may resolve the issue-

1) If any hardware device is installed, please try to remove and boot.
2) Post the PSOD error or its snap.
3) update all the devices and system ROM firmware to latest version.
4) Remove additional devices such as add on cards (HBAs), memory, disks, PSU etc. and run with minimul hardware coniguration. It it works, try booting by adding the devices one by one. so that you can understand which device is the cause, if any.
5) Open the HPE support case and update the issue.
Regards,
Shiva_JR

[AlexRie]

I just had the same issue with an DL360 Gen11, Vmware 7.0.3 and iLO 1.6.

Changed the iLO management IP, did an iLO reset and the server crashed with a PSOD.

I will investigate further on monday and open a case with HPE. 

[IT_Support_CXN]

Same issue here with DL360 g11, DL380 g11 with vSphere 8.0u3 and ILO 1.60.

Downgrading ILO is a workaround.

[daviddahles]

Same issue with a DL360 Gen 11 with vSphere 7.0u3 and ILO 1.60.

[ChrisLynch]

If you are performing an iLO Reset operation with iLO6 1.60 or iLO5 3.05 releases, we are aware of an issue causing an UMCE, resulting in the server OS crashing (aka PSOD for VMware hosts).  Initial assessment is we advise to downgrade to a previous iLO release.  We are working on a CA and an iLO firmware fix for this issue.

[Marcel_D]

I haven't seen any Customer Advisory or any public information about this. Maybe it would be a good idea to do that, since it affected 99% of our ESXi-Servers...

[ChrisLynch]

I stated in my reply that we are working on a CA and firmware fix. We are in the initial stages of triage and resolution here.

[ChrisLynch · Accepted solution]

We published an advisory on the issue of issuing an iLO Reset will cause the operating system to crash:  Advisory: (Revision) HPE Integrated Lights-Out 5 (iLO 5) and iLO 6 - Critical Advisory - Servers Running iLO 5 Firmware v3.05 or iLO 6 Firmware v1.60 May Experience Unplanned Server Outage After Initiating a "Reset iLO"

The CA outlines updating to iLO5 3.06 and iLO6 1.06, as they both contain the fix for this inadvertant bug.

[FrancWest]

We were also hit hard by this issue. It brought our entire vSphere cluster down.

Why didn't HPE sent out a critical email alert about this issue? I'm subsribed to the advisories notifications, but didn't receive one about this.

It's also not foundon the download page of Ilo 6. This is such a critical issue that customers should be informed by all possible means.