HPE DL360 Gen9 - Will not boot - no VGA. iLO pre-boot console works, no critical issues

[hpdavids]

Hey folks,

I have recently come in posession of a DL360 Gen9 that seems to be "cactus" as we call it in Australia.

When turning on, the fans spin and almost immediately a red light starts flashing at the front console. There is no VGA output at this time.

When I leave the server off and press the UID button, I get VGA output and am presented with the HPE ProLiant Pre-boot Health Summary, in which I get my iLO IP, hostname and Firmware version as well as two CPLDs. There's also a Critical Integrated Management Log Event which tells me there are No critical events.

I am able to access iLO and as long as my server is off, my Health status is OK.

When I turn on the server, VGA turns off and I get the red flashing light again. iLO however remains accessible but tells me the system status is Critical. Specifically, the BIOS/Hardware Health shows a big fat red cross.

IML doesn't show anything new (and I wiped it for good measure). It does show me that I have recently flashed a System BIOS (version P89 3.08) - Which was my attempt to get things back.

When going through the diagnostics, I noticed that in the Redundant ROM section, both the "active" and the "backup" ROMs state "Record not found".

I am assuming this means that the previous owner has tried to flash a new System ROM that went wrong, or has actively wiped this (potentially during decomissioning), but I cannot seem to make this go away.

Having access to iLO, as well as a HP SPP Gen9 iso (through my work) I have tried to create an SPP iso and boot from it without luck. I have run SUM from my desktop, added a node with the server IP (successfully) which then tells me that everything is okay and can't update anything (except for iLO, which I can force to reinstall).

Through iLO I have tried to "update firmware" of both iLO and the system ROM, which looks to work, but after reboot puts me in the same situation with no VGA and a red flashing LED.

Google tells me I may need to run the preboot UEFI console, but without VGA output, and only access to iLO this feels impossible. When trying to select the boot order through iLO I get an error message stating: "
Error fetching boot order details: SMBIOS Record 219 is not available."

I have reseated the CPUs, the RAM and the fans to no avail.

I have tried to boot with SW6 in the ON position (allegedly forcing a reset) without any luck - right now all switches are in the OFF position again.

Is it stuffed? Or is there any way that I can save this server from the scrapheap?

Cheers,

Peter

[Tam92]

Hello,

This could be a fault related to the hardware probably the system board.

We can isolate using the minimum configuration.. Use one CPU and 1 DIMM to boot the server. Remove all the additional cards.

You can disconnect the HDD Backplane cables as well in this process.

If it powers on without any errors, please add the remaining parts one by one and look for any errors.

You may also try to restore manufacturing default settings and check

https://techlibrary.hpe.com/docs/iss/proliant_uefi/UEFI_Gen9_121417/s_Restoring_Default_Manufacturing_Settings.html#:~:text=From%20the%20System%20Utilities%20screen,a%20setting%20and%20press%20Enter.

Thanks,

TAM

[hpdavids]

Thanks Tam,

I'll give it another go removing everything. As mentioned, I had already removed and reseated everything (and trying to boot at intermediate steps - not mentioned but not my first rodeo) but hey, not the first time that I've had to do the same time multiple times for it to randomly work I'll remove the second CPU and all but one DIMM as well as all other add in cards (10G network and I'll also remove the raid controller, because what's the worst that can happen - knock on wood).

As for "restore the manufacturing defaults" - The link you've provided states "Go to the system utilities screen", which will be impossible since there's no screen to go to, and I remember searching for it in iLO 4 and couldn't find it.

I'll keep you in the loop on my "remove everything" escapades. According to the documentation "there are only two valid fan configurations" one for 1 CPU and one for 2 CPUs, so I'll also remove the fans that are flagged as incorrect for a single CPU setup.

Cheers,

Peter

[hpdavids]

Hey Tam,

I am now in single DIMM and single CPU mode, as well as single CPU and without any hard drives.

I am getting the same issue still. I have swapped CPUs (ie. tried the other one), swapped PSUs, swapped DIMMs to no avail.

I can still get into pre-boot and see things on my VGA screen, but powering on turns off the screen and starts the red flashing light.

What I have noticed is that the iLO event log is saying "Server reset." about every 5 to 10 seconds (hard to say, because I took everything out I lost the time settings, and I cannot find where to set it again)... But I am not pressing anything.

[Torsten.]

Open the cover, locate the DIP switch on the board and set

S1, S5, and S6 to on

This will enable the redundant system ROM.

[hpdavids]

Thanks Torsten,

Unfortunately I am getting the exact same behaviour with SW1, 5 and 6 on (everything else is off)

[Torsten.]

Last chance: Download the BIOS in windows version, unpack it, locate the binary BIOS file and install this via the ILO interface.

[hpdavids]

If that's the ~8MB file ending in .flash, I have tried that with two different versions. One from the SPP dvd/iso, and one that I downloaded from the internet. I have tried both an upgrade to 3.08, as well as a downgrade (I believe I am on 2.96 right now)

[Torsten.]

Try to use

Software Details - Online ROM Flash Component for Windows x64 - HPE ProLiant DL380 Gen9/DL360 Gen9 (P89) Servers | HPE Support

and extract P89_3.08_01_12_2023.signed.flash

(probably the same file like the flash download).

[hpdavids]

I believe that is one of the files I have already downloaded and flashed without success, however I will try that again in the morning (it is 9:30pm on this side of the planet )

[hpdavids]

Updated System rom (with provided link) as well as iLO to the latest version. I had downgraded these earlier in an attemt to get *something* out of it.

Unfortunately, same issue with these newer versions.

Now I am wondering, am I barking up the wrong tree with the "System ROM not found" error and is something maybe significantly wrong with the board or another component?

It does weird me out that the pre-boot (power off, press UID button) does give me VGA output, but I cannot seem to interact with that at all. And then when I turn on the server (either through the power button or through iLO's simulated button press) VGA just goes off. The virtual KVM also shows "No display" when I tried that, so there definitely is something preventing output (and probably preventing the server from booting/starting/initialising *anything*).

[Tam92]

Hello,

This looks like a hardware fault to me. As all the possible troubleshooting steps have already been performed with no luck.

I would suggest getting the System board replaced for the server.

You can log a support case with HPE for further support if required : HPE Support Case Manager

Thanks,
TAM

[hpdavids]

Thanks Tam,

Silky q. How would I go about getting a replacement board? The server is out of warranty, and I do not have a support contract in place for myself, and I would not feel comfortable calling on my work's support agreement.
Do I still just raise a support ticket? Or am I going to have to buy something online?

Cheers,
Peter

[Tam92]

Hello Peter,

HPE support will be on chargeable basis.

You may also reach out to the HPE reseller for replacement support on this.

Thanks,

TAM

[BradHome1]

1. From the System Utilities screen, select System Configuration > BIOS/Platform Configuration (RBSU) > Advanced Options > Embedded Video Connection and press Enter.

2. Select a setting and press Enter.

   1. Auto—The external video connection to the embedded video controller is automatically disabled to save power when a monitor is not attached. It is enabled automatically when a monitor is attached (including when the server is operating).

   2. Always Disabled—The external video connection to the embedded video controller is disabled and a monitor connected to this port does not display except during system boot.

   3. Always Enabled—The external video connection to the embedded video controller is always enabled. This option is only required if a monitor is attached with a monitor detection that does not function, causing Auto mode not to work properly. Select this value and do not use auto with a Ilo card in teh system for it will turn off the video card every time.

3. Press F10.

[JOR2]

Hey...

Can I have the checksum for that file (P89_3.08_01_12_2023.signed.flash)?