Betreff: FW Flashing fail and no power

[dhchoi]

I've been flash wrong SPS firmware to my microserver gen10 plus by my mistake.

Exactly I've updated microserver gen10 plus V2 model's SPS firmware to normal gen10 plus model.

The update process went to infinite loop and I tried force reboot and then the green power button light just flashes and never powers on.

The iLO is working fine but I cannot reinstall new firmware due to I cannot turn on my server.

Do I need to have the system board replaced? Please help me!

[Suman_1978]

Hi,

Before going for replacement, you can try to clear the NVRAM from the System Maintenance Switch.
Refer to this guide, item# 4.  System maintenance switch descriptions, S6 is the switch.

Thank You!
I work with HPE but opinions expressed here are mine.

[shiva_jr]

Hi @dhchoi ,
    Sometime it works by iLO settings such as "Swap ROM BIOS' option and installable listed in "Remove the installation queue" if any. Please try.
Regards,
Shiva_JR
Please mark as 'Accepted solution'  and give me 'Kudos' if my post worked.

[dhchoi]

I already tried to "Swap ROM BIOS" setting in iLO, but same symptoms.

I think swap process needs to system turn-on but it's not able to turn-on the system now.

Does it needs to change DIP swtich (such as "S1, S5, S6") on the board?

[dhchoi]

OK, I'll try to change DIP switch to S6. Many Thanks!

[Suvamay]

If this does not work then System board is suspected 

[Kijan]

@Suvamay Hello Together,

I have the same problem with 2 servers.
I downloaded the wrong firmware from the HPE Support Portal under my product and noticed too late that the text refers to V2.
Unfortunately, iLO accepted and flashed the firmware without any problems. On one server it has been flashing for 8 hours...

On the other server I have already tried all the steps described above and also tried to restart the flashing with ilorest, but without success because the server cannot be switched on.

I have seen that there is a recovery image for the PLC firmware for the V2 that can be flashed via iLO web. Is there something like this for the Gen 10 Plus without V2?

[Kijan]

Hello Together,

now it's a month later with no nresponse.

I tried to use the System Recovery Set, bit this does not include the sps-firmware.

I created a new System Recovery Set with the ilorest tool.

The server started the firmwareupdate but crashes every time with the folowing error.

Firmware Update

The last attempt to update or upload firmware was not successful. Make sure you are using a valid, signed flash file and try again. If you are attempting to install a component, first upload it to the iLO Repository then add it to the Installation Queue.

I tried different files (versions of firmware and OS) everytime the same result.

Is there a special signed version of the firmware?

What I ask myself about this topic is:

•  Why am I being offered the update for Microserver Gen 10 plus V2 to the serial number of an "old" Gen 10 Plus?
• Why is the reference to V2 only in the title?

If you select the product under "Find a Version", you are also offered Microserver Gen 10 Plus (without V2)

• The server accepts the file and starts the update although the version is not intended for the device.
• Why is the "wrong" version still offered in the Support Center?

[AnarS]

I am experiencing the same issue and have similar questions. Clearly, it is HPE's fault that their support center provides an incorrect firmware version that damages the device. In my case, rather than resolving the issue correctly, they issued a service quotation of ~1560 euros to replace the board. It's more cost-effective to purchase a new server! Consequently, I will not buy from them again and will instead opt for a different manufacturer to avoid such inadequate aftermarket support.

[dhchoi]

Right!, HPE has a lot of responsibility for this issue.

They are no longer providing me with countermeasures because of terminated warranty.

Likewise, I ended up buying another new product because the board cost was very high.

I want to know method of physically re-flashing the ROM IC (via ROM copy machine) and a ROM program.

By the way, where is the SPS stored?

[Suvamay]

Can you please log a trust case with HPE and we will see what best can we do to resolve the issue 

[AnarS]

---

@dhchoi wrote:

I want to know method of physically re-flashing the ROM IC (via ROM copy machine) and a ROM program.

By the way, where is the SPS stored?

---

I hope you get the answer to your question from someone else as I don't know it, but please keep me posted if you manage to restore the board.

[AnarS]

---

@Suvamay wrote:

Can you please log a trust case with HPE and we will see what best can we do to resolve the issue 

---

Where do you think I got the service quotation from?
The best thing you can do is stop recommending this firmware in the help center until more people run into the same problem, let your R&D department reproduce the problem, which is pretty easy to do, and post a guide on how to restore the board after this (if that's even possible), but unfortunately I doubt you'll do any of that.

[dhchoi]

Please escalate this issue to your QA.

I've already opened the case, but the answer back was that I couldn't get help because of the end of the warranty.

However, I believe that we need to rescue cutomer who were victimized by HPE's fault.

[Suvamay]

I can see there are two seperate link for Gen10 Plus and Gen10 plus V2

ROM Flash Firmware Package - Server Platform Services (SPS) Firmware for HPE MicroServer Gen10 Plus

ROM Flash Firmware Package - Server Platform Services (SPS) Firmware for HPE ProLiant MicroServer Gen10 Plus v2 Servers

[dhchoi]

In my case, firmware update is impossible because the power is not on. I understand that iLO can only be updated with firmware when the power is on. Can I update the firmware without power on?

[AnarS]

---

@Suvamay wrote:

I can see there are two seperate link for Gen10 Plus and Gen10 plus V2

---

@Suvamay 

Yes, there are, but:
1. V2 is only mentioned in the title; the description states that “This firmware package contains HPE Gen10 Plus SPS firmware for HPE Gen10 Plus servers.”
2. For some reason, my V1 server is listed under "My Applicable Products (1)."
3. The server accepted this firmware despite it being for another product.

Of course, you could blame me (and others who have encountered this issue) for not paying enough attention, but considering all of the above, how easy do you think it is to make this mistake?

Lastly, why does this firmware damage systems to such an extent that not even reverting to the previous BIOS using a switch works, leaving no possibility of restoring the server?

[Moderator edit: Erased the Confidential info. ]

[Kijan]

Another week later and the update is still being offered.
I'm beginning to wonder if HPE just doesn't care or if this "error" is intentional.

@Suvamay @shiva_jr @Suman_1978
You are HPE employees, can't you escalate the issue internally?
Creating support cases doesn't seem to help as you just refer to the lack of warranty and make an absurd repair offer. (See @AnarS  post)

The issue is now 2 months old and not even the entry in the support portal has been updated.

Screenshot-1_2024.05.18

The "System Recovery Set" is also currently still an issue, unfortunately the version created at the factory only contains iLO5, System ROM & System Programmable Logic Device.

Creating a new "System Recovery Set" with the "Server Platform Services (SPS) Firmware" in version 5.1.4.913 or older does not work.
The package can be uploaded as .fwpkg or as .signed.bin in the iLO repository and you can then create a new "System Recovery Set" via ilorest, but the process always aborts when installing the "Install Set" via iLO webinterface.

Error message (red banner):

The last attempt to update or upload firmware was not successful. Make sure you are using a valid, signed flash file and try again. If you are attempting to install a component, first upload it to the iLO Repository then add it to the Installation Queue.

The following can then be observed in the "Installation Queue":

Status: In Progress

System Recovery Set item (Server Platform Services)
Name: System Recovery Set item (Server Platform Services)
Command: Apply update
Component name: Server Platform Services (SPS) E3 Firmware
File name: MicroserverGen10PlusSPS_05.01.04.913.signed.bin
State: In Progress
Result: Not available
Installable by: iLO
Maintenance Window: None specified
Start Time: Immediately after the associated updater checks
Expiration: Never
Update Recovery Set? No
Created by user with Recovery Set privilege? Yes

Status: Exception

System Recovery Set item (Server Platform Services)
Name: System Recovery Set item (Server Platform Services)
Command: Apply update
Component name: Server Platform Services (SPS) E3 Firmware
File name: MicroserverGen10PlusSPS_05.01.04.913.signed.bin
State: Exception
Result: The update failed with a component specific error. Retry the update after fixing the component error.
Installable by: iLO
Maintenance Window: None specified
Start Time: N/A
Expiration: Never
Update Recovery Set? No
Created by user with Recovery Set privilege? Yes

1. Is there a way to influence or even bypass the check before installation?
2. Why is the signature accepted during the normal installation during the boot process but not during the installation via an install set?

[Kijan]

Any further Information ? HP plays the Dead Man ?  

What to do with the 1000 bucks paperweight ?

[KW17]

First time I heard of this, stubmled a little to see some SPS 6 update for the Gen 10 Plus v2 in my list but avoided to 'try'.

Seems that HP includes (and flashes) the CSE SPS region instead of updating via Intel tools / firmware. Normally INtel CSE tools are version specific and would deny to work with different firmware versions.

One possibly could try to find the SPI chips, dump them with a programmer and injecting back the correct SPS region? Since it's bricked already there's nothing to loose.... There are some forums that seem to have experience in unbricking 'normal' firmwares like winraid or badcaps, maybe worth a try, too.

EDIT In the storagereview Microsoerver Gen 10 plus review (link) there's a pic of the mainboard with heatsink removed. In the corner beween fan and the small chipset- heatsink (towards USB2 connector / memory slots) is an chip marked "ME ROM".

[RonnyReaggy]

I do have the same situation. System is bricked, support gives no . They killed two of my systems, which are total garbarge now. Thanks for destroying my money....

[KW17]

Well, as written- there's a good chance of repair:

Download MicroserverGen10PlusSPS_05.01.04.913.fwpkg

Open the file in a hex editor

Cut/delete 0x108D in the beginning (blue marked = delete / not marked new start of file)

Cut 0x3F34 in the end, (blue marked = delete / not marked new end of file)

The file should now be precisely 0x800000 in size- save it.

Get a (quite cheap) CH341 programmer and a SOIC 8 clip

Remove the mainboard from the chassis, remove the cooler.

Read out the marked chip. You should get at least two 100% identical reads with a content similar to the file you extracted to be sure that the clamp is seated properly and that you're on the correct chip (Don't check checksums only- a badly positioned soic clamp will give you wothless 8 MB FFs with a proper looking checksum)

Flash the chip with the saved 8 MB file you extracted from MicroserverGen10PlusSPS_05.01.04.913.fwpkg in the beginning.

Don't rely on the flash- programs verify function only. Read back the chip and compare its new content with the file intended to write. The proiginal file and the re-read new content of the chip should be 100% identical.

Re- assemble the machine.

The system should work again (but it might need some reboots and take more time to boot first time(s)

(Since the system no longer boots and this part of the firmware neither can be flashed from ILO without a booting system there's hardly nothing HPE could offer after a brick except for a new mainboard)

Disclaimer:

There are several images of the mainboard on the net, they might have changed the relevant chip to a WSON type which would need another type of adapter (can't be done with a SOIC clamp)

You might damage your machine further when doing this improperly by kicking of small SMD parts when positioning the clip, or shortening pins or ..... (but it's a brick anyway, so...)

Didn't di this on my machine since I never flashed the wrong file / had to

There aren't any machine specific data in this part of the firmware before first boot so you won't need any information from the old content.

[Kijan]

Good morning everyone,

@KW17Thank you for your advice, I got the server unbricked.

I think there is no guarantee the this hint will work for you but its worth a try.

Helpful links

• https://github.com/YTEC-info/CH341A-Softwares
• https://www.amazon.com/s?k=ch341a

[KW17]

@KijanThanks for the feedback, good to hear that it worked!

[rsommefeldt]

Just a quick one to say that this resurrected my Gen10 Plus that was affected in the same way. If it helps anyone else, mine has a Macronix MX25L6473F for the ME ROM SPI flash, and I used a cheap ch341a flasher which came with a SOP8 grabber and cable adaptor.

I flashed from macOS using the flashrom command line utility. After creating the 8MiB ROM image from MicroserverGen10PlusSPS_05.01.04.913.fwpkg as @KW17 mentioned it had the sha256 sum of b92532af89a89689a9dc0c9c6a532788368f522f81d6a893d98fd2566e9c7862.

I flashed like this:

sudo flashrom --programmer ch341a_spi --chip MX25L6473F --write extracted_rom.fwpkg

Then dumped the ROM to make sure the contents matched:

sudo flashrom --programmer ch341a_spi --chip MX25L6473F --read rom_dump.bin

The sha256 of the ROM dump matched the one above, so I powered the server on from the iLO and it was fine from then on. Thanks @KW17!