ProLiant DL380 G6 Continually Reboots

[gnostos]

Hello

We have 3 proliant DL380 G6 servers in three node microsoft cluster. Each one of them reboots 3 to 4 times a day unexpectedly. Server characteristics are:

Processor: Intel X5550 2.67 64bit
RAM: 20 GB
OS: Microsoft Windows Server 2008 R2 Enterprise x64 Version, (Build 7600) – HyperV Role ( VM )
ILO Firmware Version 1.79

We have opened ticket in HP. In one of the servers we have changed motherboard with no luck. HP still tries to find out what the problem is.

We have applied last firmware, BIOS updates ( Smart Start 8.30 ).

We even applied the ms hotfix about the ACPI C3 processor state ( KB 975530 )

As you can undertand we have plenty of virtual servers in this cluster and every time a reboot occurs in one of the servers we have loss of service.

Can anyone help with this situation.

Thanks in advance

[TTr]

Unless you got a bad batch of hardware, it is unlikely that all three servers have intermittent hardware failures. Have you looked in the cluster itself? Did you find any events in each server regarding the reboot? Anything in the logs? (hardware, OS, cluster). You need to look further nad find something relating to the reboots.

[gnostos]

We have checked OS, Cluster Logs and there isn't any related event recorded. The reboot occurs suddenly and there is no telling in which of the 3 servers will occur.
Apart from that the only thing that it is the same in all 3 servers is that the last event
Recorded in the system event log is the 'winhttp web proxy auto-discovery service stopped' event. We have disabled this service with no result.

[gnostos]

FollowUp:

We have disabled ASR in all three servers with no results.

Is there any other who had similar situation with the same configuration?

Any info would be helpfull

Thanks again

[Michael Eltiste]

4 of our 11 DL380 G6 Servers are exhibiting this spontaneous reboot behavior.

All are P62 BIOS and 8.60 Firmware.

Ticket still open with HP support.
They replaced the following parts in one of the systems and it hasn't rebooted in the last 48 hours.
2 x 511778-001 AC, hot-plug, 750 W power supply
1 x 496069-001 System I/O board assembly
1 x 496062-001 Power supply backplane

I think this is a fairly widespread problem.

[JKytsi]

How about the ILO driver for windows ? Is it latest. There is an advisory about the ILO fw 1.79 and ASR reboots.

[Michael Eltiste]

Yes, we are on 1.79 iLo. We aren't actually getting ASR's though. We are running Windows 2008 x64 O/S and only get a previous shutdown unexpected message in the system log.

We are getting the same errors in the iLo log that others have noted.

Server power restored.
Server power removed.
Server reset.

This isn't a power problem, we have an enterprise class UPS supporting the entire data center. Two separate power legs delivered to each rack. No power interruptions indicated on other servers in this rack.

[gnostos]

Hello again

We have changed motherboard in two of the three servers and the haven't rebooted in the last 24 hours.

We will wait and see !!!

[John S King]

We have a similar situation with unexpected shutdowns several times a week. 2 dl380 g6 servers with 2008 R2, one local, one remote connected to a san. It's like somebody pulls the plug, there is no memory dump or event log entry or any other rhyme or reason I can see. We have redundant power supplies. we booted with smart start and all hardware diagnostics pass. we have latest firmware and drivers. HP is sending 2 system boards (I hope is the fix). We have to put these machines in production asap so I will be watching this post and respond when we come up with a solution.

[marcus1234]

this is a "known issue"provide the serial number to HP support and they will let you know if it is part of the bad batch...

but if you get event id 57 in windows or in iml log that is another issue


then refer to this
and here you need ilo firmware , and management controller then reboot


http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766〈=en&cc=us&taskId=101&prodSeriesId=3884113&prodTypeId=15351

enjoy and points would be nice :)

[JEerd]

We're experiencing the same problem. However the HP Technical Support Representative states that the 460 Watt Power Supply is not supported on DL380 G6 and we should buy the 750 Watt Power Supply. We have 3 DL380 G6 servers, only one has this problem. Can someone give us some advice on the issue? HP Benelux is unaware of this "known issue".

[Michael Eltiste]

After hours of fighting with support, they agreed to replace the motherboards in the remaining 3 servers. No reboots since system board replacement 48 hours ago.

Note: You will need to re-input your advanced iLo license if you paid for that.

The other workaround suggested by tech support was setting the "High Performance Power mode" in the BIOS. Setting the power mode through iLo is different and doesn't work. It must be done in the BIOS.

Hopefully engineering will release a permanent fix for this soon.

[Victor Kuznetsov]

I had the same problem. Could you please let me know where "High Performance Power mode" is located in the BIOS? I found only High Efficiency Mode (Auto).
Thanks!

[marcus1234]

install the latest firmware update it is not on the firmware cd then you should see it enjoy

:) points appreciated

[Joel Meyer]

I'm seeing same issue with DL380 G6. I've made the BIOS change recommended by HP (maximum performance) and reboots still occur.

[Michael Eltiste]

Victor,
Yes this setting is found in the P62 BIOS of DL380 G6 servers. I'm not sure about any other models or BIOS revisions.


Joel,
I'd pressure HP support to replace the motherboard in the affected system. That seems to have resolved the issue for us.

Good Luck,
Michael

[Yeo]

My customer also have same problem but after changed the motherboard seen working fine. Is it all the server if install window 2008 having same problem? Because I have 2 servers install window 2003 haven't got the problem.

[Joel Meyer]

We are running using ESX v4 and seeing this problem.

[mwanzi]

We recently saw this same issue as well. We swapped motherboards on several nodes, but based on the frequency of the reboots it appeared to be a systemic issue. This is a known issue, and for now it appears HP can have a tech come on site and flash some system component. We are still waiting to get more details.

[Victor Kuznetsov]

I changed power profile in BIOS to High Performance from Balanced mode and problem disappeared for a week.

[Iain Turnbull]

We had a batch of four of these servers, three of which have gone through this problem - a new mainboard fixed all three.

[TarunJain]

Did any one else tried to change power profile in BIOS?

[marcus1234]

I know i have sorted a few of these issues,

power profile works on some servers only it is part of the issue, chnging it just makes server use a bit more power ( so it is not so enviroment friendly etc...

now heck iml log in ilo look for event id 57


as i mentioned before
this is a "current issue"provide the serial number to HP support and they will let you know if it is part of the "special batch"...

but if you get event id 57 in windows or in iml log that is another issue easily fixed

for that 57 issue refer to this
and here you need ilo firmware , and management controller then reboot

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766〈=en&cc=us&taskId=101&prodSeriesId=3884113&prodTypeId=15351

enjoy and points would be nice :)

[marcus1234]

hi
This is how you thank in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

[Victor Kuznetsov]

mark1234,

Do you know the number of this issue in HP?

Russian support, unfortunately, do not know anything about this issue...