topic Re: FW Flashing fail and no power in ProLiant Servers (ML,DL,SL)

FW Flashing fail and no power

dhchoi — Mon, 25 Mar 2024 02:40:44 GMT

I've been flash wrong SPS firmware to my microserver gen10 plus by my mistake.

Exactly I've updated microserver gen10 plus V2 model's SPS firmware to normal gen10 plus model.

The update process went to infinite loop and I tried force reboot and then the green power button light just flashes and never powers on.

The iLO is working fine but I cannot reinstall new firmware due to I cannot turn on my server.

Do I need to have the system board replaced? Please help me!

Re: FW Flashing fail and no power

Suman_1978 — Fri, 15 Mar 2024 05:55:07 GMT

Hi,

Before going for replacement, you can try to clear the NVRAM from the System Maintenance Switch.
Refer to this guide, item# 4. System maintenance switch descriptions, S6 is the switch.

Thank You!
I work with HPE but opinions expressed here are mine.

Re: FW Flashing fail and no power

shiva_jr — Fri, 15 Mar 2024 06:17:49 GMT

Hi @dhchoi ,
Sometime it works by iLO settings such as "Swap ROM BIOS' option and installable listed in "Remove the installation queue" if any. Please try.
Regards,
Shiva_JR
Please mark as 'Accepted solution' and give me 'Kudos' if my post worked.

Re: FW Flashing fail and no power

dhchoi — Fri, 15 Mar 2024 07:13:52 GMT

I already tried to "Swap ROM BIOS" setting in iLO, but same symptoms.

I think swap process needs to system turn-on but it's not able to turn-on the system now.

Does it needs to change DIP swtich (such as "S1, S5, S6") on the board?

Re: FW Flashing fail and no power

dhchoi — Fri, 15 Mar 2024 06:24:50 GMT

OK, I'll try to change DIP switch to S6. Many Thanks!

Re: FW Flashing fail and no power

Suvamay — Wed, 20 Mar 2024 14:38:28 GMT

If this does not work then System board is suspected

Re: FW Flashing fail and no power

Kijan — Mon, 16 Sep 2024 09:32:13 GMT

@Suvamay Hello Together,

I have the same problem with 2 servers.
I downloaded the wrong firmware from the HPE Support Portal under my product and noticed too late that the text refers to V2.
Unfortunately, iLO accepted and flashed the firmware without any problems. On one server it has been flashing for 8 hours...

On the other server I have already tried all the steps described above and also tried to restart the flashing with ilorest, but without success because the server cannot be switched on.

I have seen that there is a recovery image for the PLC firmware for the V2 that can be flashed via iLO web. Is there something like this for the Gen 10 Plus without V2?

Re: FW Flashing fail and no power

Kijan — Mon, 16 Sep 2024 09:23:13 GMT

Hello Together,

now it's a month later with no nresponse.

I tried to use the System Recovery Set, bit this does not include the sps-firmware.

I created a new System Recovery Set with the ilorest tool.

The server started the firmwareupdate but crashes every time with the folowing error.

Firmware Update

The last attempt to update or upload firmware was not successful. Make sure you are using a valid, signed flash file and try again. If you are attempting to install a component, first upload it to the iLO Repository then add it to the Installation Queue.

I tried different files (versions of firmware and OS) everytime the same result.

Is there a special signed version of the firmware?

What I ask myself about this topic is:

Why am I being offered the update for Microserver Gen 10 plus V2 to the serial number of an "old" Gen 10 Plus?
Why is the reference to V2 only in the title?

If you select the product under "Find a Version", you are also offered Microserver Gen 10 Plus (without V2)

The server accepts the file and starts the update although the version is not intended for the device.
Why is the "wrong" version still offered in the Support Center?

Re: FW Flashing fail and no power

AnarS — Mon, 16 Sep 2024 09:28:28 GMT

I am experiencing the same issue and have similar questions. Clearly, it is HPE's fault that their support center provides an incorrect firmware version that damages the device. In my case, rather than resolving the issue correctly, they issued a service quotation of ~1560 euros to replace the board. It's more cost-effective to purchase a new server! Consequently, I will not buy from them again and will instead opt for a different manufacturer to avoid such inadequate aftermarket support.

Re: FW Flashing fail and no power

dhchoi — Fri, 10 May 2024 05:19:51 GMT

Right!, HPE has a lot of responsibility for this issue.

They are no longer providing me with countermeasures because of terminated warranty.

Likewise, I ended up buying another new product because the board cost was very high.

I want to know method of physically re-flashing the ROM IC (via ROM copy machine) and a ROM program.

By the way, where is the SPS stored?

Re: FW Flashing fail and no power

Suvamay — Fri, 10 May 2024 05:23:21 GMT

Can you please log a trust case with HPE and we will see what best can we do to resolve the issue

Re: FW Flashing fail and no power

AnarS — Fri, 10 May 2024 07:36:30 GMT

@dhchoi wrote:
I want to know method of physically re-flashing the ROM IC (via ROM copy machine) and a ROM program.
By the way, where is the SPS stored?

I hope you get the answer to your question from someone else as I don't know it, but please keep me posted if you manage to restore the board.

Re: FW Flashing fail and no power

AnarS — Mon, 16 Sep 2024 09:26:23 GMT

@Suvamay wrote:
Can you please log a trust case with HPE and we will see what best can we do to resolve the issue

Where do you think I got the service quotation from?
The best thing you can do is stop recommending this firmware in the help center until more people run into the same problem, let your R&D department reproduce the problem, which is pretty easy to do, and post a guide on how to restore the board after this (if that's even possible), but unfortunately I doubt you'll do any of that.

Re: FW Flashing fail and no power

dhchoi — Fri, 10 May 2024 07:47:20 GMT

Please escalate this issue to your QA.

I've already opened the case, but the answer back was that I couldn't get help because of the end of the warranty.

However, I believe that we need to rescue cutomer who were victimized by HPE's fault.

Re: FW Flashing fail and no power

Suvamay — Mon, 16 Sep 2024 09:25:34 GMT

I can see there are two seperate link for Gen10 Plus and Gen10 plus V2

ROM Flash Firmware Package - Server Platform Services (SPS) Firmware for HPE MicroServer Gen10 Plus

ROM Flash Firmware Package - Server Platform Services (SPS) Firmware for HPE ProLiant MicroServer Gen10 Plus v2 Servers

Re: FW Flashing fail and no power

dhchoi — Mon, 16 Sep 2024 09:24:44 GMT

In my case, firmware update is impossible because the power is not on. I understand that iLO can only be updated with firmware when the power is on. Can I update the firmware without power on?

Re: FW Flashing fail and no power

AnarS — Mon, 16 Sep 2024 09:24:04 GMT

@Suvamay wrote:

I can see there are two seperate link for Gen10 Plus and Gen10 plus V2

@Suvamay

Yes, there are, but:
1. V2 is only mentioned in the title; the description states that “This firmware package contains HPE Gen10 Plus SPS firmware for HPE Gen10 Plus servers.”
2. For some reason, my V1 server is listed under "My Applicable Products (1)."
3. The server accepted this firmware despite it being for another product.

Of course, you could blame me (and others who have encountered this issue) for not paying enough attention, but considering all of the above, how easy do you think it is to make this mistake?

Lastly, why does this firmware damage systems to such an extent that not even reverting to the previous BIOS using a switch works, leaving no possibility of restoring the server?

[Moderator edit: Erased the Confidential info. ]

Re: FW Flashing fail and no power

Kijan — Mon, 16 Sep 2024 09:29:29 GMT

Another week later and the update is still being offered.
I'm beginning to wonder if HPE just doesn't care or if this "error" is intentional.

@Suvamay @shiva_jr @Suman_1978
You are HPE employees, can't you escalate the issue internally?
Creating support cases doesn't seem to help as you just refer to the lack of warranty and make an absurd repair offer. (See @AnarS post)

The issue is now 2 months old and not even the entry in the support portal has been updated.

Screenshot-1_2024.05.18

The "System Recovery Set" is also currently still an issue, unfortunately the version created at the factory only contains iLO5, System ROM & System Programmable Logic Device.

Creating a new "System Recovery Set" with the "Server Platform Services (SPS) Firmware" in version 5.1.4.913 or older does not work.
The package can be uploaded as .fwpkg or as .signed.bin in the iLO repository and you can then create a new "System Recovery Set" via ilorest, but the process always aborts when installing the "Install Set" via iLO webinterface.

Error message (red banner):

The following can then be observed in the "Installation Queue":

Status: In Progress

System Recovery Set item (Server Platform Services) Name: System Recovery Set item (Server Platform Services) Command: Apply update Component name: Server Platform Services (SPS) E3 Firmware File name: MicroserverGen10PlusSPS_05.01.04.913.signed.bin State: In Progress Result: Not available Installable by: iLO Maintenance Window: None specified Start Time: Immediately after the associated updater checks Expiration: Never Update Recovery Set? No Created by user with Recovery Set privilege? Yes

Status: Exception

System Recovery Set item (Server Platform Services) Name: System Recovery Set item (Server Platform Services) Command: Apply update Component name: Server Platform Services (SPS) E3 Firmware File name: MicroserverGen10PlusSPS_05.01.04.913.signed.bin State: Exception Result: The update failed with a component specific error. Retry the update after fixing the component error. Installable by: iLO Maintenance Window: None specified Start Time: N/A Expiration: Never Update Recovery Set? No Created by user with Recovery Set privilege? Yes

Is there a way to influence or even bypass the check before installation?
Why is the signature accepted during the normal installation during the boot process but not during the installation via an install set?

Re: FW Flashing fail and no power

Kijan — Thu, 13 Jun 2024 18:31:47 GMT

Any further Information ? HP plays the Dead Man ?

What to do with the 1000 bucks paperweight ?

Re: FW Flashing fail and no power

KW17 — Sat, 09 Nov 2024 20:20:06 GMT

First time I heard of this, stubmled a little to see some SPS 6 update for the Gen 10 Plus v2 in my list but avoided to 'try'.

Seems that HP includes (and flashes) the CSE SPS region instead of updating via Intel tools / firmware. Normally INtel CSE tools are version specific and would deny to work with different firmware versions.

One possibly could try to find the SPI chips, dump them with a programmer and injecting back the correct SPS region? Since it's bricked already there's nothing to loose.... There are some forums that seem to have experience in unbricking 'normal' firmwares like winraid or badcaps, maybe worth a try, too.

EDIT In the storagereview Microsoerver Gen 10 plus review (link) there's a pic of the mainboard with heatsink removed. In the corner beween fan and the small chipset- heatsink (towards USB2 connector / memory slots) is an chip marked "ME ROM".