topic Re: ProLiant DL380 G6 Continually Reboots in ProLiant Servers (ML,DL,SL)

ProLiant DL380 G6 Continually Reboots

gnostos — Wed, 11 Nov 2009 12:30:13 GMT

Hello

We have 3 proliant DL380 G6 servers in three node microsoft cluster. Each one of them reboots 3 to 4 times a day unexpectedly. Server characteristics are:

Processor: Intel X5550 2.67 64bit
RAM: 20 GB
OS: Microsoft Windows Server 2008 R2 Enterprise x64 Version, (Build 7600) – HyperV Role ( VM )
ILO Firmware Version 1.79

We have opened ticket in HP. In one of the servers we have changed motherboard with no luck. HP still tries to find out what the problem is.

We have applied last firmware, BIOS updates ( Smart Start 8.30 ).

We even applied the ms hotfix about the ACPI C3 processor state ( KB 975530 )

As you can undertand we have plenty of virtual servers in this cluster and every time a reboot occurs in one of the servers we have loss of service.

Can anyone help with this situation.

Thanks in advance

Re: ProLiant DL380 G6 Continually Reboots

TTr — Wed, 11 Nov 2009 12:59:12 GMT

Unless you got a bad batch of hardware, it is unlikely that all three servers have intermittent hardware failures. Have you looked in the cluster itself? Did you find any events in each server regarding the reboot? Anything in the logs? (hardware, OS, cluster). You need to look further nad find something relating to the reboots.

Re: ProLiant DL380 G6 Continually Reboots

gnostos — Wed, 11 Nov 2009 13:11:35 GMT

We have checked OS, Cluster Logs and there isn't any related event recorded. The reboot occurs suddenly and there is no telling in which of the 3 servers will occur.
Apart from that the only thing that it is the same in all 3 servers is that the last event
Recorded in the system event log is the 'winhttp web proxy auto-discovery service stopped' event. We have disabled this service with no result.

Re: ProLiant DL380 G6 Continually Reboots

gnostos — Thu, 12 Nov 2009 07:34:22 GMT

FollowUp:

We have disabled ASR in all three servers with no results.

Is there any other who had similar situation with the same configuration?

Any info would be helpfull

Thanks again

Re: ProLiant DL380 G6 Continually Reboots

Michael Eltiste — Thu, 12 Nov 2009 20:27:52 GMT

4 of our 11 DL380 G6 Servers are exhibiting this spontaneous reboot behavior.

All are P62 BIOS and 8.60 Firmware.

Ticket still open with HP support.
They replaced the following parts in one of the systems and it hasn't rebooted in the last 48 hours.
2 x 511778-001 AC, hot-plug, 750 W power supply
1 x 496069-001 System I/O board assembly
1 x 496062-001 Power supply backplane

I think this is a fairly widespread problem.

Re: ProLiant DL380 G6 Continually Reboots

JKytsi — Thu, 12 Nov 2009 20:41:51 GMT

How about the ILO driver for windows ? Is it latest. There is an advisory about the ILO fw 1.79 and ASR reboots.

Re: ProLiant DL380 G6 Continually Reboots

Michael Eltiste — Thu, 12 Nov 2009 20:56:06 GMT

Yes, we are on 1.79 iLo. We aren't actually getting ASR's though. We are running Windows 2008 x64 O/S and only get a previous shutdown unexpected message in the system log.

We are getting the same errors in the iLo log that others have noted.

Server power restored.
Server power removed.
Server reset.

This isn't a power problem, we have an enterprise class UPS supporting the entire data center. Two separate power legs delivered to each rack. No power interruptions indicated on other servers in this rack.

Re: ProLiant DL380 G6 Continually Reboots

gnostos — Fri, 13 Nov 2009 07:17:32 GMT

Hello again

We have changed motherboard in two of the three servers and the haven't rebooted in the last 24 hours.

We will wait and see !!!

Re: ProLiant DL380 G6 Continually Reboots

John S King — Fri, 13 Nov 2009 16:45:03 GMT

We have a similar situation with unexpected shutdowns several times a week. 2 dl380 g6 servers with 2008 R2, one local, one remote connected to a san. It's like somebody pulls the plug, there is no memory dump or event log entry or any other rhyme or reason I can see. We have redundant power supplies. we booted with smart start and all hardware diagnostics pass. we have latest firmware and drivers. HP is sending 2 system boards (I hope is the fix). We have to put these machines in production asap so I will be watching this post and respond when we come up with a solution.

Re: ProLiant DL380 G6 Continually Reboots

marcus1234 — Fri, 13 Nov 2009 19:52:58 GMT

this is a "known issue"provide the serial number to HP support and they will let you know if it is part of the bad batch...

but if you get event id 57 in windows or in iml log that is another issue

then refer to this
and here you need ilo firmware , and management controller then reboot

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766〈=en&cc=us&taskId=101&prodSeriesId=3884113&prodTypeId=15351

enjoy and points would be nice :)

Re: ProLiant DL380 G6 Continually Reboots

JEerd — Mon, 16 Nov 2009 15:37:01 GMT

We're experiencing the same problem. However the HP Technical Support Representative states that the 460 Watt Power Supply is not supported on DL380 G6 and we should buy the 750 Watt Power Supply. We have 3 DL380 G6 servers, only one has this problem. Can someone give us some advice on the issue? HP Benelux is unaware of this "known issue".

Re: ProLiant DL380 G6 Continually Reboots

Michael Eltiste — Mon, 16 Nov 2009 15:52:47 GMT

After hours of fighting with support, they agreed to replace the motherboards in the remaining 3 servers. No reboots since system board replacement 48 hours ago.

Note: You will need to re-input your advanced iLo license if you paid for that.

The other workaround suggested by tech support was setting the "High Performance Power mode" in the BIOS. Setting the power mode through iLo is different and doesn't work. It must be done in the BIOS.

Hopefully engineering will release a permanent fix for this soon.

Re: ProLiant DL380 G6 Continually Reboots

Victor Kuznetsov — Tue, 17 Nov 2009 04:07:25 GMT

I had the same problem. Could you please let me know where "High Performance Power mode" is located in the BIOS? I found only High Efficiency Mode (Auto).
Thanks!

Re: ProLiant DL380 G6 Continually Reboots

marcus1234 — Wed, 18 Nov 2009 21:02:56 GMT

install the latest firmware update it is not on the firmware cd then you should see it enjoy

:) points appreciated

Re: ProLiant DL380 G6 Continually Reboots

Joel Meyer — Wed, 18 Nov 2009 21:07:09 GMT

I'm seeing same issue with DL380 G6. I've made the BIOS change recommended by HP (maximum performance) and reboots still occur.

Re: ProLiant DL380 G6 Continually Reboots

Michael Eltiste — Wed, 18 Nov 2009 21:33:41 GMT

Victor,
Yes this setting is found in the P62 BIOS of DL380 G6 servers. I'm not sure about any other models or BIOS revisions.

Joel,
I'd pressure HP support to replace the motherboard in the affected system. That seems to have resolved the issue for us.

Good Luck,
Michael

Re: ProLiant DL380 G6 Continually Reboots

Yeo — Fri, 20 Nov 2009 04:05:04 GMT

My customer also have same problem but after changed the motherboard seen working fine. Is it all the server if install window 2008 having same problem? Because I have 2 servers install window 2003 haven't got the problem.

Re: ProLiant DL380 G6 Continually Reboots

Joel Meyer — Fri, 20 Nov 2009 04:07:05 GMT

We are running using ESX v4 and seeing this problem.

Re: ProLiant DL380 G6 Continually Reboots

mwanzi — Wed, 25 Nov 2009 15:39:55 GMT

We recently saw this same issue as well. We swapped motherboards on several nodes, but based on the frequency of the reboots it appeared to be a systemic issue. This is a known issue, and for now it appears HP can have a tech come on site and flash some system component. We are still waiting to get more details.

Re: ProLiant DL380 G6 Continually Reboots

Victor Kuznetsov — Thu, 26 Nov 2009 02:25:52 GMT

I changed power profile in BIOS to High Performance from Balanced mode and problem disappeared for a week.