PSOD with vSphere 8

[stadi13]

We face PSOD with all of our DL325 Gen10 Plus v2 with vSphere 8 (different customer installations). After investigation by VMware they say it is caused by HPE ilo package which lead to a non-empty heap PSOD. After nearly two months on the case with HPE we have no solution yet. Does anyone else face this issue?

[Suman_1978]

Hi,

Can you post full error message or code?

Is this applicable for your issue? Advisory: HPE Synergy/ProLiant Gen10 Server Platforms - Systems Configured with HPE Agentless Management Service Version 11.6 (or Earlier) and Running VMware EXSi Customized HPE Images for ESXi 6.5/6.7/7.0 May Encounter a PSOD Exception 14

Thank You!
I work with HPE but opinions expressed here are mine.
HPE Tech Tips videos on How To and Troubleshooting topics

[stadi13]

[Sebasbin]

Hi,

There are no workaround or solution yet for this issue, and you need to do followup on HPE support case which already logged.

Also, you can try fresh installation by using VMware image without HPE add-on and check if any issue is facing.

[fdit]

Hi all,

we are also using HPE DL325 Gen10 Plus with ESXi 8U1 and experience the same issue.

We also opened tickets at VMware, they also said that the reason for this PSOD is the HPE iLO software component in ESXi.

So we created a ticket at HPE and all they said to us was something like that we should wait if this issue occurs more often and then get back to them.

It doesnt seem to be a individual issue but a general.

Are there any news related to this?

[stadi13]

Hi @fdit 

The issue seems to be a known issue for several models. Our ticket with HPE is still open - meanwhile they are working with VMware on resolving the issue. The issue is caused by HPE ilo package. To avoid further PSOD you can temporary remove the ilo package from the esx host. With the next patch cycle through vLCM the package will be installed again and needs to be removed manually until further notice from HPE with a solution. I don't know why HPE is not informing proactively on this.

esxcli software vib remonve -n ilo

Regards

[MichaelRi]

I have simply uninstalled the ILO driver on the ESXi and hope that HPE and VmWare will eventually manage to solve the problem.

Connect-VIServer -Server vcenter

$esxcli = Get-VMHost esx.intern.risse-wilke.de | Get-EsxCli

$esxcli.software.vib.list() | ft

$esxcli.software.vib.remove($false, $false, $true, $false, @("ilo"))

[fdit]

Thats what we did too. Even though this ist not ideal and should definitely be fixed, it is the only workaround I know of at this point.

We had couple of our hosts crashing with the same error over and over again, mostly after around 40-42 days uptime. VMware HA saved us a whole lot of trouble!

VMware support seemed very competent with their analysis, so I very much rely on that and thats the reason we uninstalled the iLO-driver.

Now it is up to HPE to solve the problem.

[Dieter_Adriaens]

We are also having this issue on our new DL325 Gen11 servers with ESXi 8.0 update 2.

We reached out to VMWare and they confirmed it was the iLO driver that was PSODding the hosts. We opened a case with HPE and it is open in het hopes it gets fixed.

In the meantime we are removing the iLO driver from the hosts to minimize the changes of it coming back.

Why isn't this an advisory for this? There are more people experiencing this. There are advisories for strange things you get when you are doing some very freaky things to the host. But not this one., which is just a straight install/upgrade.  

[FVC_DTT]

Has anyone heard anything new from HPE about this? We just upgraded our environment from vCenter/ESXi v7 to v8, which has been rock solid for 2 years, it's now randomly crashing hosts, two within a week so far. Our external support have gathered logs/dumps and are in the process of logging tickets with both VMware and HPE.

[bsawadogo]

We just got reffered to HPE support on similar issue on DL365 gen10 after months of going back-and-forth with VMware support. Anyone found a stable workaround?

[stadi13]

You can simply uninstall the iLO driver on the ESX hosts. This will stop the PSOD of occuring - only a workaround

esxcli software vib remove -n ilo

Requires a reboot of the host.

[bfkmnemonic]

Does anyone have an update on this?

[Sunitha_Mod]

Hello @bfkmnemonic,

Thank you for writing to us. 

You might want to consider creating a new topic by utilizing the ""New Discussion"" button if you have any queries, as this will not only enhance visibility compared to the old topic but also boost your chances of receiving responses from experts.

[stadi13]

Last feedback was months ago that the fix will be implemented in vSphere 8 Update 3 which has not been released yet. We see that the issue occure much less with newest iLO firmware and vSphere 8 Update 2b.

[bfkmnemonic]

Thank you for your reply..

Weird they do not take it more serious. Deviating from the image it kind of a big deal when using desired state configuration and LCM, as you cannot make configuration changes to your host when LCM baseline is not compliant.

[stadi13]

Fully agree. It took a long time until HPE took the reponsibility for acting and stop blaming VMware. Its a manual process of uninstall the ilo package after every update cycle.

[profectIS]

Thanks for this, will try that tonight.

After five months of a monthly PSOD I opened a case last month and got told to install the newest SPP for my DL325 which would fix the issue...
Of course the issue didn't reoccur immediately after the reboot and the case was closed, but now after 34 days of uptime it happened again.

Opened a new case and this time (as of yesterday) got told to delete the ilo driver and if the problem persists, I should contact ESXi support...

[bfkmnemonic]

Any news on this. Updating to 8.0 Update 3a. Are we still uninstalling HPE vibs?

VBB Ref: https://knowledge.broadcom.com/external/article/316522/esxi-host-may-crash-with-psod-with-the-m.html

[FrostByteVA1]

Side note.  Upgraded 20 hosts from 8u2c to 8u3b.  Two consistently PSOD on boot.  Looks like it's related to Sub-NUMA Clustering (on by default in Virtualization - Max Performance).  Setting workload profile to custom and disabling SNC seems to resolve it.  VMware knows about it.  Waiting to see what HPE says.

[stadi13]

Feedback from HPE is that it is resolved in vSphere 8 Update 3. Have not faced any issues with PSOD within vSphere 8 Update 3 yet.

[bulletlynn21]

I have been having this happen to various customers running DL325 Gen11 servers. HPE just posted this recently. https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00143662en_us

[bfkmnemonic]

@bulletlynn21 

Did it resolve the issue?