ProLiant Servers (ML,DL,SL)
1748282 Members
4106 Online
108761 Solutions
New Discussion

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

 
cykVM
Frequent Advisor

Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Hello HP community,

 

I already briefly discussed the issue in VMWare communities, see https://communities.vmware.com/thread/491027

But this was not really leading to a resolution.

 

I maintain a newly bought HP Proliant DL380e Gen8 server which was freshly installed in August using the HP customized VMWare vSphere 5.5 Update 1 installation ISO. After configuration the server ran fine.

Hardware data:

HP Proliant DL380e Gen8 (bought brand new in August 2014), HP SmartArray B320i storage controller, HP H222 host bus adapter (only a HP Ultrium4 tape drive connected to that), HP Intel 4port NIC 366i, 32GB RAM, 2 Quadcore Intel Xeon E5-2407

 

I'm aware that the storage controller B320i is not on the VMWare HCL but that's why I used the customnized installation ISO.

 

After HP released a new SPP and a VMWare 5.5 Update 2 ISO beginning of September I first installed the SPP during maintenance providing several firmware updates. The ilo4 firmware was updated to 2.0 some weeks before.

Afterwards I ran an upgrade installation to VMWare 5.5 U2. All went through without issues, no errors or crashes.

 

The server was running fine for some days and suddenly the first crash of VMWare happened. The PSOD displayed was similar to the one in the attachment. Error message: PCPU 0: no heartbeat (2/2 IPIs received)

 

I rebooted the server through iLo console and during the following days the server crashed multiple times with a similar PSOD, always with PCPU 0: no heartbeat (2/2 IPIs received)

 

At the time of the crash the server/VMWare was mostly idle (at night time or very early in the morning.

 

I reviewed the BIOS settings and set those according to HP recommondations for VMWare, especially referring to power management settigs.

 

But no configuration change or setting helped, VMWare kept crashing randomly, sometimes after about half a day, 2-3 days or about a week.

 

2 days ago I started deploying a new Windows VM, initial VM configuration was successful, the VM was created on the datastore and appeared in inventory. Just at power on of that VM VMWare crashed again with No heartbeat PSOD.

This was reproduceable after a reboot of the system. After the reboot the newly created VM disappeared from the inventory but was still existing physically on the datastore volume.

 

Since this happened during office hours, I was fed up with testing various BIOS settings and things in VMWare configuration and went back to VMWare 5.5 Update 1 (build 1746018 HP customized) by using the SHIFT+r altbootbank option on boot up.

 

The server runs stable without issues since then (I know only 2 days, but ...) and new VM deployment works fine with 5.5 U1.

 

I somehow suspect a kernel <-> driver error here to be the cause of the PSODs. It might be the HP 366i 4port NIC physical driver in conjunction with the virtual E1000 NICs within the VMs or even the HP hpvsa driver for the B320i Smart Array controller.

 

Anyone around here any ideas?

 

Thanks in advance for any help provided.

 

cykVM

21 REPLIES 21
Suman_1978
HPE Pro

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Hi,

 

If possible try this:

 

In BIOS, under HP Power Regulator, use HP Static High Performance Mode.
And add VMware boot flag timerEnableTSC = false
Add VMware boot flag usePCC = false

 

See if the above settings make any difference.

 

Thank You!
I am a HP employee.

Useful Links for ProLiant Servers


I work for HPE.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]

Accept or Kudo

cykVM
Frequent Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Thanks for your suggestion, Suman but I'm afraid I can't try this right now. I went back to 5.5 U1 with the SHIFT+R method on VMWare boot because it was the only quick way to get back a stable running system.

 

I might try this during next weekend.

 

 

JaaM
Visitor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Thanks, same here.

 

I was using ESXi 5.1.0 with no problems. After update to 5.5.0 U2, PSODs began to appear. As cykVM said, i am also going to test it some of theese nights or in the weekend, cause it is a production server. I also made a rollback to 5.1.0.

 

Greetings

cykVM
Frequent Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

At VMWare forums another user faced the same problem of suddenly appearing PSODs after upgrade installation to 5.5 U2. He had similar hardware to my configuration. He also upgraded from 5.1.0 to 5.5 U2 as JaaM did.

 

See user CyrilH's post here: https://communities.vmware.com/thread/491027?start=0&tstart=0

 

 

ErikV1991
Occasional Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Hi,

 

We have the same problem, we have DL360e Gen8.

Installed from scratch ESXi 5.5 U2 on SD-Card and still get the same PSOD's

 

I have checked the BIOS and the settings are HP Static High Performance Mode.

Changed the settings in VMware for timerEnableTSC = false, but i could not find the usePCC = false !?

 

I let you know if the servers get an PSOD or not.

cykVM
Frequent Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

You may set usePCC option through VMWare vSphere client by clicking on the host -> configuration tab -> Software section / Advanced configuration -> epand VMKernel (right pane) and select Boot -> scroll down until you see VMKernel.Boot.usePCC and untick the box next to it

 

Also see http://h20195.www2.hp.com/v2/GetPDF.aspx%2F4AA3-9258ENW.pdf for further information (pages 14 and onwards)

 

I think there is also a way though vSphere CLI, but it should work this way.

 

If that helps for you it would be nice to hear about any progress. I may upgrade to 5.5 U2 then.

 

 

ErikV1991
Occasional Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

I checked it, but this one is not available in my VMware, other options are.

Maybe the Hardware ?

 

Well i check in the morning if the server is still up, i let you know.

cykVM
Frequent Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

I just checked on my VMWare, also not available. Maybe it was removed in 5.5. The PDF is referring to 5.0.

cykVM
Frequent Advisor

Re: Proliant DL380e Gen8 keeps crashing (PSOD) after upgrade to HP customized VMWare ESXi 5.5 U2

Yes, was removed in 5.0 Update 2 version and defaults to FALSE now.

See last post (correct answer) in https://communities.vmware.com/thread/465241

 

"This issue has been resolved as of ESXi 5.0 Update 2 as PCC is disabled by default ..."