DL580 Gen10 Critical Fault

[EDub]

My DL580 Gen10 no longer boots. Fans do not start-up. Flashing Green & Red leds on the front panel

I can connect only via the iLO. The iLO says BIOS/Hardware Health critical. When I click into the critcal link, it shows 

iLO detected invalid Server Platform Services firmware.

iLO detected invalid Innovation Engine firmware. 

Event Class 0x20
Event Code 0x3

I tried uploding/flashing 

Name: ROM Flash Firmware Package - Server Platform Services (SPS) ...

Version: 04.01.05.002

File name: SPSGen10_04.01.05.002.fwpkg

and

Name: ROM Flash Firmware Package - HPE Gen10 Innovation Engine Fir...

Version: 0.2.3.0

File name: IEGen10_0.2.3.0.fwpkg

I also swapped out the power supplies as well

Still no change, the Server will not boot

Any help at all would be GREATLY appreciated

[Suman_1978]

Hi,

As you have performed most of the troubleshooting and updated the FW, I would suggest few more steps:
Remove any unwanted PCI cards or USB devices which are not required for Boot.
Clear NVRAM from the System maintenance switch S6.  Refer to this guide on page#137

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00128555en_us
https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00100177en_us

If none of these steps help, please log a support case with HPE Support.

Thank You!
I work with HPE but opinions expressed here are mine.
HPE Tech Tips videos on How To and Troubleshooting topics

[Terry75]

Hello,

Make any changes hardware or firmware of late?

Do you have a recovery set in the iLO respository? 

Thanks!

[EDub]

Hu Suman_1978

I uploaded the 

Name: ROM Flash Firmware Package - HPE Gen10 Innovation Engine Fir...

Version: 0.2.3.0

File name: IEGen10_0.2.3.0.fwpkg

and

Name: ROM Flash Firmware Package - Server Platform Services (SPS) ...

Version: 04.01.04.339

File name: SPSGen10_04.01.04.339.fwpkg

Added into the Installation Oueue. Should it show as Locked ?

Also set the switch dip 6 to the ON position on the system board (Shoild it be 1,2,3 & S6? or just 6 ?).

Should I power off/unplug the server from the power outlet, then re-plug to restart? the power button does nothing but blink when pressed.

Also removed the NIC card as well.

[EDub]

Hi Terry_75

The only hardware changes I made is I removed a NIC card I added. This is what is in the iLO Repository

[EDub]

I noticed the Innovation Engine firmware and the SPS firmware seems to be stuck in a "Pending" state. does not seem to flash

[Terry75]

Hello,

Hmmm, if the server will not boot you will not be able to update any firmware.

You  removed the  NIC and then starting having this issue of not booting?

If so, I would pop the lid and reseat the riser. Had an similar issue, removed riser and the server fully booted up.

Seen where it was a bad riser and also where a reseat fix the issue

Hope that helps.

[EDub]

Hello Terry

Thanks for the help

Yes, while troubleshooting I removed the NIC card. Then tried to power-on, would not boot, fans don't start

So I removed the riser, powered-off(un-plugged), waited 15 minuted, then powered back-on(re-plugged into the outlet), same result.

Then added the riser back, was sure to push down and make sure the riser was all the way in. powered-off(un-plugged), waited 15 minutes, then powered back-on(re-plugged into the outlet), same result.

Lastly, I removed all the RAM modules, waited 15 minuted, then powered back-on(re-plugged into the outlet), hoping I would get an error or a beeping anything. same result, would not boot, fans don't start

So I set the maintenance switch dip switch 6-pin to the ON position, hoping it would force a defaulting of the NVRAM. waited 15 mins, then re-plugged into outlet. Same result. Not sure what else to do. Any ideas would be greatly appreciated

[Terry75]

Hello,

Was the server up and running and just stopped one day?

Starting to wonder if there is a issue with the backplane itself or cable.

Issue with a piece of hardware with the front LEDs flashing for sure. 

Thanks!

[EDub]

Hi Terry

This happened 3 days ago, we added a NIC to the Server. Since then it's been in this state. Powered-off the server, Removed the NIC card. waited 15 minutes, then plugged the server in the outlet and powered-up the server and it's been in this state since. Replaced the power supplies, plugged into another dedicated outlet (just to make sure it was not a power or power-supply related issue), same result.

Then removed the riser, tried everything again, same result. Re-inserted the riser (but not the NIC) same result.

Flashed the HPE Gen10 Innovation Engine firmare & the Server Platform Services (SPS) firmware via iLO (since these are what the iLO critical error points to) powered-off, then powered back on. Same result. Just green & red leds flashing on front panel, fanes do not start, server does not go into POST at all, Flashed the BIOS again v2.90. and added to the Repository. Powered off, then powered on the server. same result

Any other ideas you can think of is much appreciated

[Terry75]

Hello,

Server is being a pain for sure.

Are you able to open a support ticket with HPE?

Not sure if something was knocked loose or damage during the install of nic and riser.

Could be a bad riser DIMM slot on the system board which means will need a new board.

Just a guess at this point, would be best to open a support ticket.

Confirm all cables have solid connections and all hardware firmly seated.

Thanks!

[EDub]

Hello Terry

Thanks again for your help. Looks like I will need to open a case with HP. The down side is the server support contract has expired, 

so I will need to get a quote for a support ticket. My guess is if the system board needs to be replaced, the part will probably

be very expensive. Would know of any entities that may have used parts available ? 

Thanks