Problems with Proliant DL560 Gen 8

[SomeRandomName]

We recently recieved 8 of the above listed servers. 5 of them appear to be running normally while three of them are unexpectedly shutting themselves down. The following IML is generated just before this happens.

System Power Fault Detected. System has enabled power protection and disabled power supplies. I couldn't find much information on this entry. We've been unable to get these three servers into production for the past 4 weeks because of this.

Anyone run into this before? We've replaced the PSU's, the power backplanes, rebuilt the servers to no avail.

Thanks in advance for any help you can lend.

[Suman_1978]

Hi,

You may try two things in this case:

1.  Clear NVRAM from System Board switch.

2.  Update System BIOS (Refer to this article).

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c03612934

If issue persist, better elevate to HP.

Thank You!

[THJ]

Hi,
similar error here, although on a DL380 Gen8. I wonder how you solved this issue.
Best regards
Thomas

[JoeHHH]

Hello. I encounetred with same problem like but on DL-360P gen8 server

[vancec]

On 110 brand new DL360p Gen8 systems we have had the issue on at least 4 machines.   One machine errored out even after changng MLB.

Vance

[anthony11]

The referenced article re BIOS updates was my first thought.

[SolarPower]

We just experienced the same today. HP Proliant DL560 Gen8. Purchased the server 2 months back. Anyone aware or found a solution?

Intergrated Management Log - System Power Fault Detected. System Has Enabled Power Protection and Disabled Power Supplies (XR: 14 00 MID: FF 0D F0 CC C0 FF FF 0A 0A 0C 0C 00 9C 00 04 01 03 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00)

[anthony11]

Is your firmware up to date?

[JonathanTN]

This past weekend we experienced this error on two DL360P Gen8 servers.  One went offline Friday and another went offline Sunday with the power fault message.  A technician had to replace the motherboard in both units.  In the first server it killed two 8GB memory modules.  In the second server it killed one of the two CPUs.  Fortunately we had some extra memory on site for the first server.  For the second server we pulled the one CPU and shifted all the RAM to the second CPU.  Waiting for the replacement CPU.

[MackanNYC]

I'm Getting BIOS/Hardware Health - Failed on my system

We have now had the HP tech on site, and he has replaced the power supplies, backplane, and he discovered a Frayed Cable inside the system, so it's back up for now, but just temporarily due to he didn't have the correct spare parts so it's on back order from HP. I'm investigating if this is a know issue with HP right now. 

The server was completely down, not possible to turn it on. Flashing Amber light on the server, and the powersupplies was powered up with the green light as they should.

But I could connect to the iLO interface, so some kind of power was supplied to the iLO.  But still impossible to turn the machine on.

Tried to swap the Bios with the previous version. Still didnt work. Keep you posted when I get more info.

//Mackan

[Renwod]

Hi,

I've just run into the same problem with a DL380 G8 - did you find a resolution?

thanks,

[anthony11]

Probably too late for you, but there's an SPLD update to 0x32 code on the DL380pG8 that addresses some spurious power supply failure scenarios.

[bgrigsby]

Has anyone had any luck with HP Support in coming up with a fix for this issue?  We have had several 360p Gen8 systems that have experienced this error/behavior. Fortunately, our impact thus far has only been that the system shuts itself off, and we are able to power it back up with no damage to other components. I need to find a way to stop these outages.

Can anyone provide a link for this SPLD update?  Only reference to SPLD updates I can find is for G6 systems...

Thanks

[jmhanna]

Add me to the list, same problem here with one of two DL560 G8's purchased spring 2013.  One has been fine, the other has been resetting itself with the message:

"System Power Fault Detected. System Has Enabled Power Protection and Disabled Power Supplies"

Started in June of 2013, and we had the system board replaced then, as well as clearing the IML log and updating the BIOS.  Seemed fine up until December 27, when it happened again, and now again today.  Just opened a case (again), we'll see how that goes.

[jmhanna]

Ok, after 2 days of waiting, HP's answer (like it always is) was to update the BIOS firmware(s) with the latest SPP and clear the NV RAM (AGAIN).  Kind of frustrating, because that exactly what they had me do back in June of 2013 when this server was only a couple weeks old.  When those steps failed to resolve the issue they swapped the motherboard.  That fixed it for about 6 months.  Now the SAME problem is back, here we go again.

Does anyone at HP think this could be a POWER SUPPLY problem?  Is there potentially another fix??  Is there some way to turn off the dynamic power management stuff?  The time delay between us here in the U.S. and wherever in the world the support people are makes troubleshooting very frustrating as well.  

[Torsten.]

>> HP's answer (like it always is) was to update the BIOS firmware(s) with the latest SPP


In general it could be better to check the web for more current (and maybe fixed) firmware then the current SPP contains (this helped me with an ILO issue some weeks ago).

[anthony11]

ISTR that SPP (or whatever it's called now) has an ability to freshen components from HP's FTP server.

[PZel]

For the DL380pGEN8 there was a Advisory c04042380 (just search it with google).

Installation:  Extract the cp file and install the .vme file via ILO/Administration/Firmware/<Browse> .Then, you ve got to reboot, and look into ILO/Information/Diagnostics whether CPLD0-PAL0 is 0x32.

It don't mention anything about the DL580pGEN8 or DL360pGEN8,  however.

[Oscar A. Perez]

An excessive drop in voltage detected by either the Power Microcontroller or CPLD can trigger this Event.

[jmhanna]

**Update**

After clearing CMOS data with "switch 6" and updating the requested firmwares (BIOS, Storage controller, Power management, ILO) the server still reboots itself with the error:

EVENT (23 Jan 20:13): System Power Fault Detected. System Has Enabled Power Protection and Disabled Power Supplies (XR: 14 00 MID: FF 0D F0 0C C0 FF FF 0A 0A 0C 0C 00 9C 00 04 01 03 15 00 00 00 00 00 00 00 00 00 00 00 00 00 00)

Now HP wants me to clear CMOS again (switch 6) and then remove the systemboard battery, short those pins together, and reinstall the battery.  I'll play along, but I'm thinking it still could be a power supply issue.  

[secured2k]

DL380P G8

Same problem with the iML reporting System Power Fault Detected. System Has Enabled Power Protection and Disabled Power Supplies.

I have reset the NVRAM (Switch 6), pulled the battery, swapped the Power Supply Backplane, and disconnected most any other devices (all add card). The problem still occurs.

I attempted to use a new PSU in slot 2 and the system wouldn't power back on, but when I tried it in slot 1 (with nothing in slot 2), the system powered up. I ended up trying both PSUs in slot 1 with nothing in slot 2 and the system seemed OK.

BIOS/iLO/CPLD were all updated to the latest versions and a quick insight diagnostics and test run were made. All tests passed.

However, the problem came back about 8 hours later.

We are now trying to replace the system board.

If this doesn't work, I think it might be a power supply issue as well and will probably replace both with new ones at the same time.

[kktse0002]

I have a new DL380p Gen8 server with the same problem. The server shutdown almost every hour with a IML log "System Power Fault Detected. System Has Enabled Power Protection and Disabled Power Supplies".

I have replaced power supply, on-line UPS and CPU.

Does anyone have a solution?

[Nicolai Rasmussen]

I know this is an old thread, but we're still seeing issues that are quite similar to the ones described so far.

We've had similar issues on BL460c Gen8 and now also on DL380p Gen8.

The BL460c Gen8 actually said itself in the IML:

System Power Fault Detected. Replace System Board (XR: 10 20 MID: FF CD FC D7 03 13 13 AA 00 02 00 EE 02 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00)

So the board was replaced. This was 1 month ago and so far so good.

I've also seen a BL460c Gen8 server that just powered down without ANY information in the IML what so ever.

Via ILO i could see the lovely error "BIOS/Hardware Health - Failed", but no indications as to what was wrong. The blade refused to power up, so I tripped the E-fuse and the blade came back online, now without any indications of an error.

This was a few months ago, and I've not seen this problem since.

Until last week, where one of our most critical DL380p Gen8 servers power down with the same error: "BIOS/Hardware Health - Failed"

Power cycle cleared the error, server came back online, no errors. AHS logs sent to HP - they claim they can't see ANYTHING indicating an error...

REALLY REALLY frustrating HP. - How can we trust the Gen8 platform, when weird sh*t like this keeps happening?

And everytime you contact support, they treat you like you just graduated from high school, and act like they've never heard of this error before...

[stefanost67]

Hi all,

we have the same problem with 2 of 2 Proliant DL560 Gen 8 servers we just bought.

Let us see what support will tell us next week!

[jmhanna]

Recently HP made a change to the documentation for the latest ILO firmware update (1.51) and it sounds like there is possibly a fix in there (NOT just pertaining to a hypervisor OS, as originally reported) it now states :

"1.51 Fixes:
- Intermittent Non-Maskable Interrupt (NMI) Events May Occur on HP ProLiant Gen8 Servers running HP Integrated Lights-Out 4 Firmware Versions 1.30, 1.32, 1.40 and 1.50."

Intermittent NMI events?  Possibly.  The management log did report a power problem for every event, however the ILO log simply reported "server reset".  

I am hopeful this new firmware is the fix.  My only other 560 G8 server is running ILO 1.20, it never did the automatic reboot thing, I will leave it alone since it's not an affected version.  This problematic box was first running 1.32 then I updated it to 1.40 when instructed by support back in January.  

Fingers crossed (still)