ProLiant DL380 G6 Continually Reboots

[gnostos]

Hello,

We have changed all three motherboards and the problem still occurs.

HP support in greece does not know anything about this issue yet.

[marcus1234]

if after board replacement ensure you are not using 8 gb ram sticks

if they do not know of fix tell them to escalate immediately, then the solution will be forthcoming :)it is possible level 1 and all level 2 are not aware of fix....

[Runninjock]

I have 47 of these systems. I updated 17 to the latest BIOS (P62). That was a BIG mistake. After doing that, the random reboots started. However, I have now set the boxes power profile to "Maximum Power" and that has stopped the reboots. HP Support said their engineers were aware of the issue and a fix should be issued in the next BIOS update. Just FYI in case that helps anyone else.

[franka]

Has anyone seen this issue with DL380 G6 running Linux?

[marcus1234]

Hmm franka

this is a hardware firmware issue not really Os related


This is how we thank in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

[franka]

I suspect that it's a hardware issue as well, but I had to toss the question out there as I suspect that I might face the question as well. I've put a support ticket to HP on this but have heard literally nothing on this (usually get something saying that somebody's working on it.)

[John S King]

We gave up over a week ago and i was so hopefull! we had system boards replaced on two of these boxes, 16 days of running solid then bam, boom goes the dynamite, she blew up. i've changed the power setting in the bios since but it's too late for us, just too risky so we have to put off upgrading until mid april (it's the nature of our business). this thread started nov 11 and we talked with HP way before that and knew there was a problem. Mission critical servers shutting down for two months and not a solid fix, that's ridiculous HP

[fabio pergola]

hello...

i updated the bios rom to the latest , i also set the profile to max, but still get the unexpected shutdown...

any1 with a solution?

I already supplied HP with the .cab file of my system (report)

But i am not happy with : we are working on it... i need a solution or atleast a temporarely solution

Thnx

[TarunJain]

Hi Fabio,
Try changing power profile in BIOS.

As per POST by Victor Kuznetsov
Nov 26, 2009 02:25:52 GMT Unassigned
I changed power profile in BIOS to High Performance from Balanced mode and problem disappeared for a week.

[Joel Meyer]

We've had a case open for nearly a month and have gone through many many troubleshooting steps including running vm-support and cfg2html with no answers from anyone. After our sr. tech exchanged some words with HP a replacement motherboard is on its way from Hawaii. Hopefully this will be a permanent fix. We have 1 bad of the 4 dl380 g6 servers we ordered together.

[marcus1234]

okay folks
they are not faulty boards per say but some may have issues ..

power profile works on some servers only it is part of the issue, changing it just makes server use a bit more power so it is not so green etc..

also check iml log in ilo look for event id 57

and if you get event id 57 in windows or in iml log that is another issue easily fixed

for that 57 issue refer to this
and here you need ilo firmware , and management controller then reboot

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766〈=en&cc=us&taskId=101&prodSeriesId=3884113&prodTypeId=15351

if iml log only reports server shutdown and unexpected reboots , and they do not mention any device

then call HP they have a flash fix :)

enjoy :)

[fabio pergola]

HP changed motherboard, seems its fixed now

2 days without a reboot

[EagelsNestSL]

Same here. from 5 mins to 2hours reboots. Only log in iLO:
server reset
power removed
power restored

Even no OS on server, brand new box. Updated to latest ROM, iLO, RAID, nothing helps...

[Kopp2]

Hi all,
this problem is discussed with HP. This is "known issue", but no "realy" solution exists atm. (internal note mmr_na_0257880).
HP recommends us:
- update the BIOS from P62(Jun09) to P62(Oct09)
- update SmartArray p410i FW to 2.50
- change following settings in BIOS:
Reboot the server and press F9 to enter RBSU.
Select "Power Management Options."
Select "Advanced Power Management Options."
Select "Minimum Processor Idle Power State."
Select "C1E" or "No C-States."

I did it on 5 DL380 G6 and additional to changed "HP Power Profile" to "Maximum Performance".

3 days stable till now...
Regards,
Cop

[Joel Meyer]

HP changed out board and no reboots going on three days now.

[iceman76]

We had/have the problem on 2 DL380G6 with SLES 10/11 installed. The servers booted every 2-4 hours, no log entries, etc.

It showed up that hp service isn't as good at it used to be. We have 24/7 carepacks on both systems but hp was not able to replace the mainboard for more than one week. They had no spare parts available in germany or the netherlands. Finally we got two mainboards, but one was DOA. And replacing the mainboard did not help at all. The problem was still there - even worse. After fighting for another week with hp-service we managed to get a full server replacement for both systems.

When asking the guy from the hotline if the replacement servers will have the same problem he said : i will cross my fingers...

At the same time he admitted that there is indeed a problem with a revision of the mainboards. In some cases the above mentioned power settings in bios will help, in other cases a special software upgrade is required.

Attached you can find the statement he gave to us.

[EagelsNestSL]

fix is not public. Is cheaper to change IO board, then send the fix? LOL

[EagelsNestSL]

Motherboard replaced. Everything seems to be OK now... no reboots till now.

[Arian van der Pijl]

Same here. from 5 mins to 2hours reboots. Only log in iLO:
server reset
power removed
power restored

Got the same errors on 2 ESX4i servers fully patched.

[Martin Kölbl]

Looks like a fix.
http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?locale=en_US&objectID=c01955503

[marcus1234]

hmmmmmmmmmmmmmmmm,, I told you guys almost 2 weeks ago



okay folks
they are not faulty boards per say but some may have issues ..

power profile works on some servers only it is part of the issue, changing it just makes server use a bit more power so it is not so green etc..

also check iml log in ilo look for event id 57

and if you get event id 57 in windows or in iml log that is another issue easily fixed

for that 57 issue refer to this
and here you need ilo firmware , and management controller then reboot

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01802766〈=en&cc=us&taskId=101&prodSeriesId=3884113&prodTypeId=15351

if iml log only reports server shutdown and unexpected reboots , and they do not mention any device

then call HP they have a flash fix :)

enjoy :)

[Stephen Delaney]

I'm having the same issue with my DL380 G6. I notice from my iLO that I do have a serial number prior to the one specified in the hotfix link, so I suspect I am a victim too.

However I can't even get an OS installed on it as it keeps failing when checking for FS integrity on install. (ESX 4.0)

I did do a bit of R&D and found that if you have a P410i controller - you will suffer from awful IO performance on your local storage if you dont have the battery pack. (also this is for ESX 4.0 only).After reading all the comments I feel resigned to just getting a replacement rather than chance on running the hotfix.

[Gary Theimer]

I'm seeing apparent problems with the G6s. We've purchased two DL380 G6s. Both have had motherboard problems several months after purchase.

One of the servers quit working and refused to boot until the motherboard was replaced. The other is down - has been down for 6 days - even though we a have 24x7x4 SLA. The RAID keeps burning drives and losing its configuration. We're running these in separate network centers, separate UPS power conditioning.

Is there a problem with the hardware? We're about to purchase another 380 G6 but I'm starting to look at alternatives.

[Mark Ostroski]

HP has issued an advisory and flash update for this issue:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01955503&dimid=1004688314&dicid=alr_dec09&jumpid=em_alerts/us/dec09/all/xbu/emailsubid/mrm/mcc/loc/rbu_category/alerts

[Danny Arroyo]

We are having the same problem with 4 DL380 G6's. After reading this thread, I noticed that we have the same messages in our ILO Log (Power Restored....) that are indicated in this thread. Prior to reading this thread, we haven't done any troubleshooting in respect to these messages. I'll use the information on this thread and possibly HP to do further troubleshooting.

Just an FYI for those of you having this problem, we have been troubleshooting this problem with HP for the past month or so. HP made the determination that our servers reboot because of a known HP bug in the P410i controller with firmware v1.66 and 512MB cache module. During periods of high I/O (ex..backups) our servers either hang or reboot. We have also lost faith in HP tech support as they have not provided any solution to our problem and don't give an ETA on the permanent fix. All they offer is a workaround that hasn't seemed reliable either. The text below is from an email that I received from an HP tech on 12/1/09:

-------------------------
This issue is a known issue by engineering already (no confidential/public advisories on this yet) and there is a firmware that will be release to fix this.

The problem is with firmware /cache/raid configuration (512 cache decrease the performance and the 2.50 patch provided is not solving this issue yet). Lock ups will be address by the new firmware and in the meantime recommended configurations are:

1-rev 1.58/1.66 on both controllers to avoid errors (as it was to about error but can experience lock ups)

2-newer revisions without the 512 controller cache to avoid lock ups

The first is the most recommended workaround till the firmware is release and if possible leave it working without the controller cache, but till now there is no actual fix.

As I mention, engineering is already working on this and it will be release in a month or so.
------------------------

HP has "graciously" sent us 4 256MB cache modules for the P410i controller (free of charge) which I used to downgrade the controller memory from 512MB to 256MB. Although this seemed to work (along with other setting we applied to the Novell Netware OS) it doesn't explain why the first DL380G6 that we built is up and running without a reboot for about 2 weeks with the 512MB cache module.

For now our problem seems to have been resolved (we have to wait a few days to see what happens) however the inconsistencies in the resolution don't offer any peace of mind. We are still awaiting HP's permanent fix, but in the meantime, I will look into the information in this thread.

Thanks for the info.