Latest iLo5 for Synergy compute modules?

[PatrickLong]

I see that there is a new v 2.12 version of iLo 5 firmware available as of January 23rd.  However, it is not clear to me if this version can be applied to Synergy compute nodes or not?  Generally in the past I have applied new iLo firmware releases across our entire DL, BL, and Synergy environments, but this new release needs some clarification.  For example, the Revision history tab shows version Version:2.10 (20 Dec 2019) and newest version Version:2.12 (23 Jan 2020), but very conspicuously absent from that list is version 2.11, which ironically is the *only* version available on the latest HPE_Synergy_Custom_SPP_2019.12.20191219_Z7550-96800.iso.

So is v 2.11 a special release that is ONLY applicable to a subset of HPE systems like Synergy, and thus the reason it is missing from the full iLo5  history tab linked above? Or is the fact that 2.11 is missing from the history tab simply an administrative error?  Or was 2.11 previously on the history tab - but subsequently recalled due to an issue and then made to not appear in the history tab by changing the release date to Jan 1, 3000 as HPE has done in the past for other recalled firmware?  And as a followon is new version v2.12 ONLY applicable to DL, SL, etc. form factors and NOT applicable to Synergy?  Or can anyone confirm that it is safe and recommended o load new version 2.12 on my Synergy compute nodes outside of the normal Synergy custom SPP process - by using ilorest, for example?

[Mansoor1 · Accepted solution]

Hi Patrick,

Thank you for your query on firmware update on Synergy compute nodes.

If you specifially looking to update iLO across ML, BL and Synergy nodes - as far as these computes have iLO-5, it can be applied to them.

In synergy environemnent we have HPE Synergy 480 Gen10 and other computes that has iLO-5 and yes you can download and install the upgrade.

Now coming to the firmware upgrade methods.

One can perform updates using Standalone downloads or using SPP bundles.

Since the SPP bundles are released in cycles, they may not contain the most updated iLO firmware. Hence our customers have options to either update the firmware using SPP bundle or Standalone link.

For more inforamtion and comparison on SPP bundles, please refer to the following URL: https://techhub.hpe.com/eginfolib/synergy/sw_release_info/C.5.00.01_IS.5.00.01.html

Please feel free to get back to this thread for further queries and also advise closure if you think I have answered your questions.

[DanCernese]

The latest ProLiant ilO5 2.12 does not work on Synergy

[DanCernese]

Do not download and install the latest iLO5 version on Synergy.   It is never supported to download components outside the HPE Synergy Custom SPP (except for elevations from our engineering response team to serious issues).

ProLiant iLO5 2.12 does not work fully and correctly on Synergy

[PatrickLong]

So now I've got around 50 Synergy 480 and 660 compute modules to revert from 2.12 to 2.11   @DanCernese are you saying that the Synergy iLo5 binary update file included on Synergy Custom SPP is materially different from the iLo5 binary update downloadable from the support site or the regular Gen9/10 SPP? Are you able to elaborate on exactly what does not "work fully and correctly" with 2.12 on Synergy? I'm not questioning the veracity of your statement, I'm only going off my personal experience that I have not had any issues to date - which could simply be mere serendipitous coincidence.

If I'm way off base and they are in fact somehow actually different, why is it not programatically prevented to apply the ProLiant version of the iLo5 firmware to a Synergy iLo5 chip, just like if an admin incorrectly attempts to load a BIOS update for a different system type, for example? It would seem a simple check to make prior to running the update via iLoREST or internal to the iLo. If I go to the support site and search simply "iLo5" I get results for all the released versions - and NONE of them specify a subset of hardware for which that particular download file is valid...which seems like an important thing to document if certain versions don't work with certain product lines.

And yes, I am aware of the HPE "Only use the Synergy Custom SPP to update components" directive, which was a valiant goal for HPE Engineering at the dawn of the Synergy product line - but Intel's SCA-mageddon microcode rev onslaught (resulting in 9 BIOS releases for Synergy compute nodes in the past 13 months, if you include the 2 almost-immediately recalled releases 2.00 and 2.20) sort of up-ended that apple cart IMO. Unless the process to validate updates and cut new Synergy Custom SPP's starts getting agile enough to deal with the frequency with which I see "Critical"-rated updates for Synergy components released on the HPE Support site, I'm not sure how that goal can be viably be maintained.

Anyway while writing this I started reverting my first SY480 from iLo5 2.12 to 2.11 so cue the Benny Hill music

[DanCernese]

Thanks for your perspective, I share some of the same concerns.    Let's get a few things out..

Our support organization will do everything they can to assist with issues. When they are out of ideas, they're going to insist you match the supported Synergy recipe custom SPP.  There's good reason for that, it's what gets the full validation suite of all features (some of which take weeks to run, for example).

Certainly, the iLO5 chip is the same, as is its firmware.  However, it sits on a different system board, different mezz adapter card family and it has to talk to all those server devices which don't exist on ProLiant.. ..so there's Synergy specific features that must work correctly.   We are working to be more diligent to remove statement of support for Synergy when that validation has not yet occurred (and either the iLO support pages will change or the next version that works for all adapters, memory, and CPU options will get released and supercede it.

To deal with Intel, back in 2018(?) we allowed immediate update of BIOS ROMs.  So the facts are that the HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur.  Anything else is risky.  We were working to incorporate flexbility a few components at a time but then we hit issues with precisely the two components we wanted to allow open updates for.  So lesson learned, just in time you could say.

Longer term, all this will change late this year.  Stay tuned and please realize the valiant start would have been retired the instant it was no longer necessary but that time has not arrived.  Even with our CNAs you'll often find that updated driver/firmware will work for you until you try to use something beyond basic networking (such as FCoE or VPLAG/downlink LAGs, etc.).  When we publish support, it must be for the entire feature breadth across all currently supported options and even if 2/3 our customer base isn't using what's broken we can't let that go out.

Edit:  

• Customer Notification a00050817en_us.

[PatrickLong]

@DanCerneseThank you very much for this clear, concise, reasoned explanation.  I'll take your line "HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur. " as gospel going forward for my Synergy environments, which (other than the extraneous iLo update) are all set at the latest Synergy Custom SPP 2019.12.20200125 .  In the meantime, I'll set about the task of back-rev'ing the iLo's on those systems from 2.12 to 2.11.

Cheers, Patrick

[PatrickLong]

@DanCernese  I see a new standalone download Gen10 iLo5 release v2.15 dated April 9th, 2020.   Interestingly, two of the five listed Fixes in this version specifically mention Synergy:

-The HPE Synergy 3820C 10/20Gb CNA loses flexed sub-ports after a cold boot or E-fuse reset.
-iLO is temporarily unresponsive or iLO resets on HPE Gen10 servers and HPE Synergy Gen10 compute modules running HPE Infosight.

and one of the three Enhancements is specifically for Synergy:
-Users can now restore the last power state after a power outage on Synergy compute modules.

I'm honestly not trying to be provocative, but is your guidance still "the HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur", or is iLo now moved a relaxed cadence like the BIOS?  You did mention in your last response that "Longer term, all this will change late this year." but I'm not sure if this is what you meant by that.  And if your guidance remains the same, can I infer that we are very close to a holistic Synergy Custom SPP for April/May 2020 that will include this new 2.15 version for iLo5 - and it just hasn't made its way through the publishing pipeline just yet?  My Synergy node iLo5's are feeling a bit long in the tooth at version 2.11 dated Nov 12 2019 (which is the latest available on the latest Synergy Custom SPP Version 2019.12.20200326) when my rack-mounted Gen10's get to use a fully-supported 5-months and three-releases-newer v2.15

[DanCernese]

Hello @PatrickLong ,

Yes those are Synergy fixes and they are incorporated into the soon to be HPE Synergy Custom SPP 2020.05.01 awaiting web publication.  The backend IT and supply chain publication is in process to release that to the web this month.  That will, of course, bring with it the whole firmware baseline and OneView 5.2.

We have requests from customers in the same situation who want those same iLO5 fixes without requiring update to their existing infrastructure.  This is a serious and well-recognized request in product management.  However, the newest iLO5 also drags along with it a requirement to use a fixed HPE SUM, and therefore the validation process becomes complex (read: expensive).  Suffice to say we are evaluating if and how we might release the iLO5 update either by itself or against existing infrastructure.. ..but today that's only a discussion.  

[PatrickLong]

It looks like the HPE Synergy Custom SPP 2020.05.01 has now been published with the release of OneView Composer 5.20.00

As indicated, iLo5 21.5 is included in that new custom SPP along with a number of other updates. 

Thank you @DanCernese

[PatrickLong]

@DanCerneseIs the official HPE position for Synergy that the iLO firmware continues to be supported only at the level that is included on the latest released Synergy SPP?  Earlier in this thread you mentioned HPE Support was "evaluating if and how we might release the iLO5 update either by itself or against existing infrastructures" - has that idea gotten any traction internally? I note that you mentioned this is already the guidance for BIOS releases for Synergy per Customer Notice a00050817en_us

For example, Service Pack for Synergy, Sy-2020.07.02 includes HPE iLO 5 v2.18 Jun 22 2020 - which is five Critical/Recommended/Optional releases behind at this point.  Perhaps there is a new SPP for Synergy on the near horizon?  I'm rasing this issue again because the latest iLo 5 update 2.33 released Dec 18th is rated Critical and my Security team is asking me to revisit this discussion.

[PatrickLong]

I guess I will bump thiis topic again Is there any change in HPE's position with respect to the possibility of decoupling iLo from strict adherence to Synergy SPP as was done for BIOS in 2018?

The current HPE Synergy Service Pack 2021.02.01 includes HPE Integrated Lights Out 5 v 2.33 Dec 09 2020, yet both newer versions available since then include Critical rated fixes for Synergy:

Version:2.41 (26 Mar 2021) states:

Upgrade Requirement:
Critical - HPE requires users update to this version immediately.
and includes a Synergy-specific fix - Incorrect MAC address shown in iLO for Synergy 4820C network adapter

Version:2.44 (7 May 2021) states:

Upgrade Requirement:
Critical : For Synergy servers.
Recommended : For ProLiant, Apollo and Edgeline servers.

IMO it continues to be a mixed message for HPE to rate a newer iLo release Critical for Synergy servers, but then also hold the position that customers should not apply newer versions outside of the currently-available HPE Synergy Service Pack bundle. Not trying to be a pain, just trying to understand the reasoning.

[DanCernese]

"reasoning." == resources.

There has been no movement on decoupling (like Server ROM) the firmware updates for iLO on Synergy.

However, starting later this week selected and specifically identified versions of iLO4 and iLO5 will be supported for hotfixing on Synergy against identified SPPs.    

[PatrickLong]

@DanCernese  Returning to this thread just to point on that while iLo5 v 2.60 is listed on the Hotfix page for Synergy (meaning it is supported to install outside of a new SSP release), the latest release 2.65 is NOT listed as an approved Synergy Hotfix.  I understand the reasons for that, but I do want to point out the inconsistency that the iLo 2.65 Release Notes shows that Synergy Compute modules are supported - which I believe is currently inaccurate (but will eventually become accurate either by inclusion in the next SSP or via specific Synergy hotfix approval) information.

Devices supported

This version of the iLO 5 firmware is supported on the following server models:

.  .  .

• HPE Synergy 660 Gen10 Compute Module

• HPE Synergy 480 Gen10 Plus Compute Module

• HPE Synergy 480 Gen10 Compute Module

.  .  .

[DanCernese]

I am not aware of a request to support iLO5 2.65 as a hotfix.. ..maybe it will happen soon.  However, the next HPE Synergy Software Release goes right past 2.65 to a future version, so it won't be supported as part of that next bundle.

[PatrickLong]

@DanCernese  Revisiting this topic again given the recent release of iLo5 firmware 2.71 (rated critical with multiple security fixes) - do you have any indication of when some version of iLo5 newer than Version: 2.60(a) (20 Jan 2022) will be officially supported on Synergy 480 and 660 models via either:

A) adding some newer version of iLo5 firmware as a supported Synergy hotfix, that is approved to be applied outside of the set of defined and approved contents of the latest SSP (as 2.60(a) was),  or

B) an altogether new SSP release that includes a newer iLo 5 firmware. I have to assume a new SSP is forthcoming soon, based solely on length of time since last SSP release 2022.02.01?

Again, I will point out that SY480 and 660 models are included in the new iLo 5 2.71 firmware download page Compatibility dropdown list of systems - despite guidance in this thread and others to only apply to Synergy systems releases of firmware/drivers that are included in a an SSP, OR on the supported Synergy hotfix list, OR any new Synergy BIOS release. 

This apparent contradiction likely continues to be confusing to the average sysadmin looking to maintain currency of this component unless they are following this thread and others in the HPE Community Forums, or are well-versed in the standing rules of Synergy firmware application. In my opinion, it would be clarifyingly helpful if the Synergy compute module models were not included in the Compatibility list for any iLo 5 version past 2.60(a) - until those newer versions are in fact fully approved and supported to run on Synergy models by either the hotfix or the new SSP methods listed above. I know the certification process for Synergy is a much more involved process than for ProLiant, in part due to Synergy's more tightly-coupled components - and as a result it takes longer.  And I also suppose it is somewhat a matter of symantics - Compatible does NOT strictly mean the same thing as Approved or Supported.  Maybe what I am asking is not possible due to the limitations of how the Support Site updates are published (i.e. if the Compatibility list is not applicable on a per-version basis, but rather a single compatibility list for all released versions.)  But it certainly is confusing to keep seeing the Synergy models appear in the Compatibility dropdown of the 2.63 (Recommended), 2.65 (Recommended), 2.70 (Optional), and 2.71 (Critical) iLo releases when in fact the latest supported iLo 5 firmware that we should be running on our Synergy systems per available hotfix guidance is the much older 2.60(a) version.  

[DanCernese]

There is a  plan to release the next SSP later this month that includes iLO 5 version 2.71 [subject to change without notice].

The strategy is exactly what you indicate-- the iLO verion(s) are not supposed to list Synergy as supported until they are supported.  Our execution is lacking.  Likewsie in the HPE CVE documentation, I did notice they followed that strategy-- the Synergy servers are not listed, yet.

[PatrickLong]

Once more into the breech on this topic...  I was searching today for iLo5 firmware and I see that iLo 5 v2.72 was released on 9/14./022. Now, I know well enough from previous discussion to NOT load any new iLo firmware on Synergy compute modules uunless they are either explicitly included on an SSP .iso OR they are listed as an approved hotfix - which 2.72 is not

The interesting thing is that I can no longer see the Synergy-approved 2.71 version that *is* included on the latest 2022.08.01 SSP anywhere on the HPE support site.  Did the iLo 5 v2.71 get recalled for some reason? And if so what does that mean for Synergy compute nodes that are currently running it?

[DanCernese]

https://support.hpe.com/connect/s/softwaredetails?softwareId=MTX_a1318b7c192f42db8d24af3017&language=en_US ?

i went to support.hpe.com and searched for iLO 5 software and then used the ilo page's revision history

[PatrickLong]

@DanCernese  Fair enough.  It did not come up for me when searching directly for iLo5 2.71, although searching directly for iLo5 2.65 and iLo5 2.72 did yield results linking to those versions.  However, when I search now, all three of those versions menioned above show direct links to those versions, without having to go to the latest 2.72 and then clicking version history.   In any case, I appreciate the clarification - thank you.