DL360Gen10 system ROM/BIOS update

[Jiri3]

Hi guys,

I succesfully installed HPE server one year ago.  Default system ROM version was  U32 v2.02.  Subsequently  updates 2.10 to 2.16 worked perfectly by ILO Update Firmware utility . So last version i have today.. ILO5 is version 2.16.

But updates to 2.20 or 2.22 or 2.34 without success.  ILO show message: 
"The last attempt to update or upload firmware was not successful. Make sure you are using a valid, signed flash file and try again. If you are attempting to install a component, first upload it to the iLO Repository then add it to the Installation Queue."

So, I tried IloRest by fwpkg U32 v2.22 successfully. But after server reboot I have only version 2.16 again.

Table ILO Information shows:
System ROM U32 v2.16
Redundant System ROM U32 v2.22

Table ILO Firmware&OS Software shows:
System ROM U32 v2.16
Redundant System ROM U32 v2.10

Could me to say, where is version U32 v2.22 after reboot?  Or what is current and redundant/swapped version?

Thanks  Jiri

[BPSingh]

Hello Jiri,

Greetings!

The current and redundant ROM information can be checked from iLO under Information-iLO Overview page.

If you find the system ROM still not updated, please perform an iLO reset from Information-Diagnostics page and re-attempt the flash process using "Update Firmware" option in iLO.

Have you also tried flashing the BIOS from Operating System?

Regards,
Bhupendra

[Jiri3]

Hallo Bhupendra,

thank you very much for your respond.

I tried again:
- reset ILO and update to new version 2.18 at the same time,
- try flash process using "Update Firmware":
files U32_2.22_11_13_2019.signed.flash and also U32_2.34_04_08_2020.signed.flash
- result the same Message: "The last attempt to update or upload firmware was not successful...."

Please, let me know what is mentioned by you "flashing from Operating System"?
It is (putty) connection, for example "ssh 16.1.15.1"?
and command "load /system1/... -source http://16.1.15.2/..."

Unfortunately I do not know the exact syntax for this utility...

Regards

Jiri

[BPSingh]

Hello Jiri,

Greetings!

What Operating system is the host running?

1. Please update the HPE Innovation Engine Firmware (if not updated) from iLO web browser. You can verify the current firmware version from firmware tab in iLO.

Download link: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_641abebae642483b823c2dcc23&swEnvOid=4168#tab2 

If IE version is 0.1.4.4 (or later) please use flash file IEGen10_0.2.2.0.signed.bin

IE version 0.1.3.2 (or earlier)then use IEGen10_0.2.2.0B.signed.bin

Reboot Requirement: Reboot is required after installation for updates to take effect and hardware stability to be maintained.

2. Also please update the iLO firmware to latest before re-attempting BIOS update from iLO interface.

Regards,
Bhupendra

[Jiri3]

Hello Bhupendra,

our host server Operating System is linux, debian/buster.

Regards

Jiri

[BPSingh · Accepted solution]

Hello Jiri,

Greetings!

May I know if you had performed the steps/workaround shared in the my previous update?

If you continue to see the same issue, please download online version of system ROM as per OS flavour installed and attempt to flash the BIOS. Installation steps are mentioned on the download link itself.

https://support.hpe.com/hpesc/public/km/product/1010093150/HPE-ProLiant-DL360-Gen10-Server-Models?ismnp=0&l5oid=1010007891#t=DriversandSoftware&sort=relevancy&layout=table&numberOfResults=25&hpe=1

[Jiri3]

Hallo Bhupendra,

thank you for your respond.

I understand that two steps are needed:
1. Update the HPE_Innovation_Engine_Firmware to 0.2.2.0 from ILO_web browser
2A. Update system ROM on U32_2.34_04_08_2020.signed.flash from ILO_web browser
OR
2B. Update system ROM on U32_2.34_04_08_2020.signed.flash from POST, -> F9 -> System Utilities....

This requires shutting down the respective reboot server. Unfortunately,  we operate our server in a continuous
7x24 service and we must announce a service break at least two weeks in advance. So I assume I will be able
to try out the recommended steps in early August.

Thank you very much for your help and time.

Jiri

[BPSingh]

Hello Jiri,

Greetings!

Please follow the blow in sequence.

1. Update iLo firmware from iLo web interface (server re-boot not required) and re-attempt to flash BIOS from iLO.

2. If issue persists, please update HPE Innovation Engine Firmware (if not updated) from iLO web browser and then try to flash BIOS update from iLO interface.

Note: Reboot is required after installation for HPE Innovation Engine Firmware updates to take effect and hardware stability to be maintained.

3. Please try to update system ROM from iLO rest as mentioned under installation instruction no below link.

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_ee6001463ecb4ffc8544070363#tab1

4. If you are still unable to flash BIOS from iLO interface and iLO reset from above steps. Then please attempt to update system ROM online from the operating system level. Do no see online ROM update option for Debian/Ubuntu, please check if you can use another lInux flavour file like RHEL, CentOS etc.

https://support.hpe.com/hpesc/public/km/product/1010093150/HPE-ProLiant-DL360-Gen10-Server-Models?ismnp=0&l5oid=1010007891#t=DriversandSoftware&sort=relevancy&layout=table&numberOfResults=25&hpe=1

5. Another option is to use Gen10 Service Pack for ProLiant (SPP) ISO which is 2.32 ROM version for DL360 Gen10. You can boot from the ISO and perform the update.

https://techlibrary.hpe.com/us/en/enterprise/servers/products/service_pack/spp/index.aspx#!&tab=TAB1

6. If any assistance required with ROM update during the downtime, please reach us by selecting any of the options under "Contact Support" on the web-link below.

https://support.hpe.com/hpesc/public/home

[Jiri3]

Hallo Bhupendra,

my report after downtime:

First, I found out via the POST screen that the backup ROM had already been updated to the newest release 2.34.
POST shows this:
System ROM U32 2.16
Redundant System ROM U32 2.34
Apparently this is the result of my previous attempts. A month ago, I successfully uploaded this version 2.34
and even then I saw that it would not turn into current.

Then I went to update:
ILO firmware updated and reset to newest version 2.18 - success
Innovation Engine updated to newest version 0.2.2.0 - success
Server Platform Service updated to newest version 4.1.4.339 - success
Firmware information see image below.

Restart the server and then:
Flash BIOS update via ILO web browser - unsuccessful
Flash BIOS update via ILOrest - unsuccessful
Flash BIOS update via SPP - unsuccessful

So I think that the BIOS update works, but only on the backup ROM and will not allow it to be
overwritten same version or does the switch backup to curent after flash not work? The update
process failed in the flash phase, there will be no reboot yet.

My questions:
- Is there any way to safely overwrite the existing backup ROM?
- Would it be possible to downgrade - to version 2.16, ie the same as on the current ROM and then again to update?
- How would the update behave in POST - F9 - Embedded Applications - Firmware update?
- Would it be possible to manually try to switch the backup/current ROM in RBSU?

BTW:
I'm trying to look for errors in some log, but I can't open the AHS log. Viewer writes a message:
Error: File was not able to be processed as SMBIOS type 0 records are not found.

Thank you very much for your help.

Jiri

[BPSingh]

Hello Jiri, 

Thank you for the update.

I understand that the backup ROM got updated to 2.34.

In that case, please toggle the system ROM from iLO.

Replacing the active system ROM with the redundant system ROM Page#92

https://itpfdoc.hitachi.co.jp/manuals/ha8000v/hard/Gen10/iLO/880740-002a_en.pdf

You may refer Integrated Management logs to check if any errors are getting logged. If you want to have the AHS logs analysed then I would advise you to open a support ticket.

- Is there any way to safely overwrite the existing backup ROM?
Updating BIOS should update the primary ROM automatically.
- Would it be possible to downgrade - to version 2.16, ie the same as on the current ROM and then again to update?
Yes, we can swap ROM and downgrad but again it is not recommended. Instead make the latest ROM as primary.
- How would the update behave in POST - F9 - Embedded Applications - Firmware update?

Updating the firmware from the UEFI Embedded Shell

Procedure:

Access the System ROM Flash Binary component for your server from the Hewlett Packard Enterprise Support Center (https://www.hpe.com/support/hpesc).
When searching for the component, always select OS Independent to locate the binary file.
Copy the binary file to a USB media or iLO virtual media.
Attach the media to the server.
Boot to the UEFI Embedded Shell.
To obtain the assigned file system volume for the USB key, enter map –r.
Change to the file system that contains the System ROM Flash Binary component for your server. Enter one of the fsx file systems available, such as fs0: or fs1:, and press Enter.
Use the cd command to change from the current directory to the directory that contains the binary file.
Flash the system ROM by entering fwupdate –d BIOS -f filename.
Reboot the server. A reboot is required after the firmware update in order for the updates to take effect and for hardware stability to be maintained.

- Would it be possible to manually try to switch the backup/current ROM in RBSU?
You can swap from iLO as shared above.

[Moderator edit: Updated the broken link.]

[Jiri3]

Hallo Bhupendra,

thank you very much for your respond.. The backup ROM is really 2.34. It shows POST and linux query too:
:~# hwinfo --bios | grep -A1 ROM
ROM Size: 16384 kB
Features: 0x0f03000000007dc9da80
--
String 1: "System ROM"
String 2: "v2.16 (09/12/2019)"
--
String 1: "Redundant System ROM"
String 2: "v2.34 (04/08/2020)"

But the ILO on the Firmware & OS Software page shows the old version 2.10 - see image below. I tried ILO restart repeatedly, but no success..  It will swap ILO to ver. 2.34 work?

Thank

Jiri

[BPSingh]

So on the POST and OS you see 2.34 as the backup ROM.
iLO still shows 2.10

Please swap ROM from iLO and check the result, a reboot would be required for the changes to take place.

For iLO reporting incorrect info, please try Clearing the Active Health System Log Page 61

https://itpfdoc.hitachi.co.jp/manuals/ha8000v/hard/Gen10/iLO/880740-002a_en.pdf

If the issue persists, perform a complete power cycle (by removing the power connections) of the server.

[Moderator edit: Updated the broken link.]

[Jiri3]

Hello Bhupendra,

I cleared AHS log and restarted ILO (no server). I have one change::  ILO on site Information shows backup ROM 2.34 too.

I'll sumarize it: 
ILO Information:                              backup ROM 2.34   see image bellow
ILO Firmware&OS Software:           backup ROM 2.10    see image bellow
</>hpiLO-> show /system1/swid3   backup ROM 2.10

OS linux                                            backup ROM 2.34
POST                                                 backup ROM 2.34

is it possible to find this data somewhere?

I swapped ROM from ILO  - see image bellow.  Is it possible to find some change somewhere?
Server reboot  can I can try only at night, about 5 Hours CET or 20 hours PDT.

-------

AHS log viewer show again:
Error: File was not able to be processed as SMBIOS type 0 records are not found.

[Moderator edit: Erased the confidential info.]

[Jiri3]

Hallo Bhupendra,

after all, only one change after the swap:. See image bellow - Installation queue...

What does the red mark and "Exception" mean?

Jiri

[BPSingh]

Hello Jiri, 

Exception—The task could not complete. Subsequent tasks will not run until this task is removed from the queue.

Please refer "Queued task summary" details on page# 116 on the iLO user guide.

As you have swapped system ROM from iLO, please reboot the server for the changes to take place.

Regards.

[Jiri3]

Halo Bhupendra,

Thank you for the explanation. What is interesting for me is that the task is named after File name: U32_2.34_04_08_2020.signed.flash. So no doubt about version 2.10 according to which I entered the swap.

As for the server reboot: I won't get permission to exclude faster than Thursday morning, maybe even later.

I'll get back to you, thank you so far!

Jiri

[Jiri3]

Hallo Bhupendra,

ll try to toggle it tonight, between 8 - 9 p.m. Pacific TZ. Is there a big  risk that it will not work and the server won't boot?

Jiri

[BPSingh]

Hello Jiri,

Swap ROM option from iLO will only swap the active ROM and redundant ROM and ideally this process shouldn't cause any issue until the redundant ROM is corrupt.

In case of any issues, please revert back the original ROM (swap again from iLO) and reboot the server.

Regards.

[Jiri3]

Hallo Bhupendra,

unfortunately, don't toggles. I tried to repeat, once the Reset button and a second time Coldboot. I am sending printscreen.

Version 2.16 again. 

Jiri

[Moderator edit: Erased the confidential info]

[BPSingh]

Hello Jiri,

Thank you for the update.

There seems to be some underlying issue which needs to be investigated.

Please generate new AHS logs and log a support ticket.

Please refer "Contact Support" on link below:

https://support.hpe.com/hpesc/public/home

[Jiri3]

Hello Bhupendra,

thank you for your patience!

Jiri

[YasirLatif]

Downgrade D8000 enclosure firmware 111 to 0105. Download the firmware file from the following link

https://internal.support.hpe.com/hpesc/public/swd/detail?swItemId=MTX_ef255b981e214dd2af6cc307cd#tab2

Make sure you run the ./setup with option -g if you like to downgrade the version. 

Note: 16:51:03.216 - If you want to downgrade, please re-run SC with the -f/-g option.
16:51:03.216

Once you install the firmware package you can find your setup script under this path.

/usr/lib/x86_64-linux-gnu/firmware-d8000-0105-1.1

[root@DL360G10K21U16 firmware-d8000-0105-1.1]# ls -al
total 26412
drwxr-xr-x 2 root root 319 Sep 26 16:50 .
drwxr-xr-x 5 root root 80 Sep 26 16:50 ..
-rw-r--r-- 1 root root 10542561 Sep 9, 2019, 93-00001181_hpe_cffebod_dual_path_combined_fw_inband_r2018.36.2.gff
-rw-r--r-- 1 root root 10542625 Sep 9 2019 93-00007488_hpe_cffebod_single_path_combined_fw_inband_r2018.36.2.gff
-rwxr-xr-x 1 root root 5783496 Sep 9 2019 ccissflash
-rw-r--r-- 1 root root 457 Sep 9 2019 Config.xml
-rw-r--r-- 1 root root 32831 Sep 9 2019 CP041302.xml
-rw-r--r-- 1 root root 448 Sep 9 2019 .cpq_package.inc
lrwxrwxrwx 1 root root 5 Sep 9 2019 hpsetup -> setup
-rw-r--r-- 1 root root 1043 Sep 9 2019 payload.json
-rwxr-xr-x 1 root root 3237 Sep 9 2019 .setup >>>>>>>> do not use this
-rwxr-xr-x 1 root root 6904 Sep 9 2019 setup              >>>>>>>>>>>>>>>>>>>>>>>>  choose this script file do not use "dot" (.setup) use only setup file
-rw-r--r-- 1 root root 113612 Sep 26 18:31 soulapi.log

Note: Make sure you give executable permission.

chmod +x setup (Normally file always has executable permission)

./setup -g

then yes

This process will take about 15 minutes 

18:13:34.609 Checking Storage device at Port: 2E
18:13:34.611 Found 1 flashable SEPs
18:13:34.611 Inspecting SEP information:
18:13:34.611 Discovered: D8000
18:13:34.611 - Unit has ROM 0111
18:13:34.611 - The unit doesn't require a firmware update
18:13:34.611 Clean config detected
18:13:34.611 Applicable Seps List
18:13:34.611 The Seps present in SepFwVer 0111
18:13:34.611 Final Seps List For Flash
18:13:34.611 The Seps present in SepFwVer 0111
18:13:34.611 --------------------------------------------------------------------------------
18:13:34.611 ################################################################################
18:13:34.611 Applicable devices:
18:13:34.611 ################################################################################
18:13:34.611 HPE Smart Array E208e-p SR Gen10 in slot 1:
18:13:34.611 Port: 2E Box: 1 Device: D8000 Firmware version: 0111
18:13:34.611 ################################################################################
18:13:34.611 The Smart Component was unable to retrieve all the hardware information from the Storage Array.
18:14:01.303 Preparing to flash devices on HPE Smart Array E208e-p SR Gen10 in slot 1...
18:14:01.305 Found 1 flashable SEPs
18:14:01.305 Inspecting SEP information:
18:14:01.305 Discovered: D8000
18:14:01.305 - Unit has ROM 0111
18:14:01.305 - The unit doesn't require firmware update
18:14:01.307 Iterating through FW image: 93-00001181_hpe_cffebod_dual_path_combined_fw_inband_r2018.36.2.gff
18:14:01.307 Dualdomain : 0 , SingleDomain: 1
18:14:01.307 Iterating through FW image: 93-00007488_hpe_cffebod_single_path_combined_fw_inband_r2018.36.2.gff
18:14:01.307 Dualdomain : 0 , SingleDomain: 1
18:14:01.307 Single domain configuration detected. Loading FW Image: 93-00007488_hpe_cffebod_single_path_combined_fw_inband_r2018.36.2.gff
18:14:01.307 ROM is 10542625 bytes
18:14:01.360 ROM Image was 10542625 bytes from disk
18:14:01.360 Found SEP version : 0111
18:14:01.360 Flash set uniqueID: AC:437757367,S:377, rev: 111.000000 for flashing
18:14:01.360 Setting FLASH TYPE: SERIAL
18:14:01.360 Disable reset of SEPs...
18:14:01.360 Requesting flash for SEPs in port: 2E - this could take up to 15 minutes...
18:28:13.767 - The SEP flash operation succeeded
18:28:13.767
18:28:13.767 Pause background activity on PORT: 2E
18:28:13.767 Disable HOTPLUG EVENTS on PORT: 2E
18:28:16.737 - The SEP pause background activity operation succeeded
18:28:16.737
18:28:16.738 Restarting the Device... D8000: AC:437757367,S:377 Port: 2E Box: 1
18:28:16.917 - The SEP Reset operation succeeded
18:28:16.917
18:28:16.917 Waiting for SEPs to come back: 3 Minutes
18:31:16.917 Resume background activity on PORT: 2E
18:31:16.917 Disable HOTPLUG EVENTS on PORT: 2E
18:31:56.680 - The SEP resume background activity operation succeeded
18:31:56.680
18:31:56.680 Perform Requery opertion on seps
18:31:56.687 - The SEP Requery operation succeeded
18:31:56.687
18:31:56.687 ************* Post flash validation ************
18:31:59.228 Expected firmware version: 0105

18:31:56.680 Perform Requery opertion on seps
18:31:56.687 - The SEP Requery operation succeeded
18:31:56.687
18:31:56.687 ************* Post flash validation ************
18:31:59.228 Expected firmware version: 0105
18:31:59.228
18:31:59.228 Find SEP with WWID: 500C0FF051F8883E
18:31:59.228 - found
18:31:59.228 - firmware version is 0105
18:31:59.228 - firmware version matched
18:31:59.228 ************************************************
18:31:59.228 Flash completed Successfully behind HPE Smart Array E208e-p SR Gen10.
18:31:59.228 --------------------------------------------------------------------------------
18:31:59.228 --------------------------------------------------------------------------------
18:31:59.228 Flash Summary: #######################################################
18:31:59.228
18:14:01.303 HPE Smart Array E208e-p SR Gen10 in slot: 1
18:31:59.228 Port: 2E - SEPs Flash Successful.
18:31:59.228 - Flash completed Successfully.
18:31:59.228 ###################### Device flash Successful #######################
18:31:59.228
18:31:59.228 HPE Storage Enclosure ROM Flash: Completed with return code 0: 26 Sep 2023 - 18:31:59
18:31:59.228 --------------------------------------------------------------------------------
18:31:59.228 ================================================================================
18:31:59.228 Session completed: Tue Sep 26 18:31:59 2023