ProLiant DL360e Gen8 | VMware 5.5u2 HP Custom Nov14 | Random Purple Screens/Hypervisor Crashes

[SpudGunzZz]

Hello All,

I have a DL360e Gen8 server that is randomly crashing (screen shots attached) -sometimes after a week, sometimes after a day. Open a ticket with HP and VMware and I'm not getting anywhere. VMware support tried even capturing a dumpfile with no luck and beleive it to be a hardware related failure. I'm wondering if anyone has had a similar problem like this and might have additional options to try.

The only simptoms I can relate to the Hypervisor crashes is they appear when a large I/O operation is taking place or has taken place (e.g. snapshot removals etc.). The first crashed appear during the initial migration of VM's: Veeams quick migration tool and VMware standalone converter tool. Random crashes now appear during or after Veeam Nightly backups. Not sure if this helps. In addition, iLO logs do not show any signs of problems as well.

Right now I'm in the process of re-installing the Hypervisor to see if it might have been from a bad ISO.

Thanks for any help in advance!

--

ProLiant DL360e Gen8

Firmware Information:
HP Dynamic Smart Array B320i Controller: 4.50
HP Ethernet 1Gb 4-port 366i Adapter: 1.61, 0x80000c6f, 1.475.0
HP ProLiant System ROM: 08/02/2014
HP ProLiant System ROM - Backup: 08/02/2014
HP ProLiant System ROM Bootblock: 03/05/2013
iLO: 2.03 Nov 07 2014
Intelligent Provisioning: 1.61.45
Power Management Controller Firmware: 3.3
Power Management Controller Firmware Bootloader: 2.7
Server Platform Services (SPS) Firmware: 2.1.7.E7.4
System Programmable Logic Device: Version 0x11

VMware ESXi 5.5 update 2 HP Custom ISO (November 2014)

[omnix-mm]

Are you try these BIOS setting ?

Power Management : Maximum Performance.
Power Management - advance: Collaborative Power Control: Disabled.
http://h20628.www2.hp.com/km-ext/content-webapp/document?docId=emr_na-c03564134

[SpudGunzZz]

Thanks for the reply. I set both BIOS settings like you mentioned. I will let you know the results.

One thing the HP support tech noticed was the memory modules were populated in 1,2,3,5 (slot 4 was skipped) on both CPU's. We are going to take the server down tomorrow to re-arrange to 1,2,3,4. Not sure if you'd think this would cause the purple screens or not.

[omnix-mm]

Good luck to you !

I had often Memory trouble many Proliant on high speed CPU (upper intel X5600) servers.

It's made trouble the C state, Power save mode was lost memory's need power. (default is C6)

"Maximum Performance" is turn to "No C state",It's no problem.

[SpudGunzZz]

Changed both power options in the BIOS, changed the configuration of the memory to 1,2,3,4 (verses 1,2,3,5), and re-installed the hypervisor off a new ISO image -no luck.

The hypervisor crashed within the first 2 hours of a Veeam "active full" backup during the snapshot removal.

[omnix-mm]

Hmm...

If you made ESXi and datastore into internal disk on B320i, I/O performance was so heavy. I think about it.

If you can, Do you try separate external ESXi bootable flash media (ex. USB memory) and datastore on internal disk?

What Do you use hpvsa varsion (B320i VID driver ) now ?

SPP HOT FIX - HP ProLiant Gen8 Servers - DRIVER UPGRADE REQUIRED to Avoid NMI in Linux and VMware on HP ProLiant

RESOLUTION
To ensure an NMI does not occur when the server is under heavy load with low memory, upgrade to the HP Dynamic Smart Array Controller Driver as follows:

For VMware vSphere 5.5 (hpvsa) upgrade to Version 5.5.0-86.0 (or later)

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c04235232&DocLang