HP DL Servers (G5 & G7) ESXi 4.1 - Sudden Reboot

[JJT]

ESXi 4.1 (721871)

vCenter 4.1

HA and DRS Cluster (DRS set to Aggressive)

Dell iSCSI SAN (Using NC522 SFP 10Gbe)

All BIOS, NIC, etc Firmware updated to latest versions

We have had intermittent issues with some of our HP DL 360 G5 & DL 380 G7 servers rebooting all of a sudden.

The iLO logs and ESXi logs do not seem to point to anything specific and the most recent reboot did not leave a crash dump file on the ESXi host.

I have been working with Vmware and HP but could not find a definite answer. Vmware says to check with HP and HP says to check with Vmware....

Today's sudden reboot was completely out of the blue as nothing was going on in the cluster (e.g. snapshot adding/removal, storage vmotion, etc) and all of the HP server firmware is updated to the most recent versions as well as the ESXi hypervisor.

I have attached the iLO entry that shows the reboot - it looks like the power was removed but I confirmed that nobody was working in the rack at that time.

Has anyone every run into this issue?

[Jags_21]

Hi,

It seems something wrong from hardware point of view, but cannot just say hardware it could be something to do with software point of view as well.

Configure the remote syslog on these servers so that the logs are saved on a different server and can be analyzed if the issue happens again, as ESXi does not retain the logs if the server is rebooted, follow the below KB article to configure.

http://kb.vmware.com/kb/1016621

Thanks,

Jagadish

-----------------

Was the post useful? Click on the white KUDOS! Star on the left side of this post.
For HP Guided Troubleshooting; http://www.hp.com/support/hpgt

[JJT]

Jagadish - thank you for your reply.

Actually, I am alreadying using a Syslog server - vMA to be exact. I sent the logs to VMware and confirmed that up to the time of the sudden reboot, no errors or problems showed up on the logs.

I ran the HP Offline Diagnostics on the problem server but did not find any problem either.

Any other things I could check?

[Jan Soska]

Hello,

Could you explain deaper if all your server face random reboot issue? If not, is there iny difference between rebooting one's a good one's?

Could you double check your network adapter firmware and driver version? Keepn on mind there was critical problem between HP 10Gb NIC's and Vmware, check advisory: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542 and vmware drive page at https://my.vmware.com/web/vmware/details/esx4x_qla_nx_nic_dt/ZHcqYmRAdyViZHdlZQ== .

Other option: Do you have possibility to upgrade problematic host to ESXi5 (with latest drivers) and check the stability?

Jan

---

@JJT wrote:

Jagadish - thank you for your reply.

Actually, I am alreadying using a Syslog server - vMA to be exact. I sent the logs to VMware and confirmed that up to the time of the sudden reboot, no errors or problems showed up on the logs.

I ran the HP Offline Diagnostics on the problem server but did not find any problem either.

Any other things I could check?

---

[Jan Soska]

Another hint: Check this:

http://bizsupport2.austin.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02496982&lang=en&cc=us&taskId=101&prodSeriesId=3935417&prodTypeId=3709945

I think your issue could be related to combination of firmware and driver of your NC522 Qlogix adapter.

Jan

[JJT]

Jan - thanks for your reply.

Unfortunately, we were not experiencing the PSOD and we had updated the ESXi drivers before updating the firmware.

Just to update everyone - the original DL 380 G7s have been removed and replaced with DL 360 G7s with slightly different hardware. We have not experienced a sudden reboot yet but these servers have only been in production for about two weeks now. I will keep my fingers crossed...

For those that are curious, I have attached screenshots of the NIC model name and versions.

NetXen HP NC522SFP - In the servers that experience the sudden reboot.

ServerEngines Corp. NC552SFP - In the new servers and haven't experienced a sudden reboot...yet.

(MAC Address column removed in screenshots below)

[Jan Soska]

Hello,

This is pretty strange. Your adapters are for sure totally diferent - they use diferent drivers, firmwares etc..

These are onboard devices? Or aditionals?

If I check dl380 g7 quickspecs - it show HP NC382i onboard devices - again another type.

Jan

[JJT]

Jan - thanks for your reply.

On the G7s, we have four onboard 1 Gb NICs, 1 PCIe 4 x 1Gb NIC card, and the dual-port NC552SFP card for iSCSI.

The G5s are similar but only two onboard 1Gb NICs and slight different quad-port NIC cards and dual port NC522SFP card for iSCSI.

So far, no sudden reboots but this is what happened last time - just when we thought things were okay, ZAP! :) I'm keeping my fingers crossed....