Latest iLo5 for Synergy compute modules?

[PatrickLong]

I see that there is a new v 2.12 version of iLo 5 firmware available as of January 23rd.  However, it is not clear to me if this version can be applied to Synergy compute nodes or not?  Generally in the past I have applied new iLo firmware releases across our entire DL, BL, and Synergy environments, but this new release needs some clarification.  For example, the Revision history tab shows version Version:2.10 (20 Dec 2019) and newest version Version:2.12 (23 Jan 2020), but very conspicuously absent from that list is version 2.11, which ironically is the *only* version available on the latest HPE_Synergy_Custom_SPP_2019.12.20191219_Z7550-96800.iso.

So is v 2.11 a special release that is ONLY applicable to a subset of HPE systems like Synergy, and thus the reason it is missing from the full iLo5  history tab linked above? Or is the fact that 2.11 is missing from the history tab simply an administrative error?  Or was 2.11 previously on the history tab - but subsequently recalled due to an issue and then made to not appear in the history tab by changing the release date to Jan 1, 3000 as HPE has done in the past for other recalled firmware?  And as a followon is new version v2.12 ONLY applicable to DL, SL, etc. form factors and NOT applicable to Synergy?  Or can anyone confirm that it is safe and recommended o load new version 2.12 on my Synergy compute nodes outside of the normal Synergy custom SPP process - by using ilorest, for example?

[Mansoor1 · Accepted solution]

Hi Patrick,

Thank you for your query on firmware update on Synergy compute nodes.

If you specifially looking to update iLO across ML, BL and Synergy nodes - as far as these computes have iLO-5, it can be applied to them.

In synergy environemnent we have HPE Synergy 480 Gen10 and other computes that has iLO-5 and yes you can download and install the upgrade.

Now coming to the firmware upgrade methods.

One can perform updates using Standalone downloads or using SPP bundles.

Since the SPP bundles are released in cycles, they may not contain the most updated iLO firmware. Hence our customers have options to either update the firmware using SPP bundle or Standalone link.

For more inforamtion and comparison on SPP bundles, please refer to the following URL: https://techhub.hpe.com/eginfolib/synergy/sw_release_info/C.5.00.01_IS.5.00.01.html

Please feel free to get back to this thread for further queries and also advise closure if you think I have answered your questions.

[DanCernese]

The latest ProLiant ilO5 2.12 does not work on Synergy

[DanCernese]

Do not download and install the latest iLO5 version on Synergy.   It is never supported to download components outside the HPE Synergy Custom SPP (except for elevations from our engineering response team to serious issues).

ProLiant iLO5 2.12 does not work fully and correctly on Synergy

[PatrickLong]

So now I've got around 50 Synergy 480 and 660 compute modules to revert from 2.12 to 2.11   @DanCernese are you saying that the Synergy iLo5 binary update file included on Synergy Custom SPP is materially different from the iLo5 binary update downloadable from the support site or the regular Gen9/10 SPP? Are you able to elaborate on exactly what does not "work fully and correctly" with 2.12 on Synergy? I'm not questioning the veracity of your statement, I'm only going off my personal experience that I have not had any issues to date - which could simply be mere serendipitous coincidence.

If I'm way off base and they are in fact somehow actually different, why is it not programatically prevented to apply the ProLiant version of the iLo5 firmware to a Synergy iLo5 chip, just like if an admin incorrectly attempts to load a BIOS update for a different system type, for example? It would seem a simple check to make prior to running the update via iLoREST or internal to the iLo. If I go to the support site and search simply "iLo5" I get results for all the released versions - and NONE of them specify a subset of hardware for which that particular download file is valid...which seems like an important thing to document if certain versions don't work with certain product lines.

And yes, I am aware of the HPE "Only use the Synergy Custom SPP to update components" directive, which was a valiant goal for HPE Engineering at the dawn of the Synergy product line - but Intel's SCA-mageddon microcode rev onslaught (resulting in 9 BIOS releases for Synergy compute nodes in the past 13 months, if you include the 2 almost-immediately recalled releases 2.00 and 2.20) sort of up-ended that apple cart IMO. Unless the process to validate updates and cut new Synergy Custom SPP's starts getting agile enough to deal with the frequency with which I see "Critical"-rated updates for Synergy components released on the HPE Support site, I'm not sure how that goal can be viably be maintained.

Anyway while writing this I started reverting my first SY480 from iLo5 2.12 to 2.11 so cue the Benny Hill music

[DanCernese]

Thanks for your perspective, I share some of the same concerns.    Let's get a few things out..

Our support organization will do everything they can to assist with issues. When they are out of ideas, they're going to insist you match the supported Synergy recipe custom SPP.  There's good reason for that, it's what gets the full validation suite of all features (some of which take weeks to run, for example).

Certainly, the iLO5 chip is the same, as is its firmware.  However, it sits on a different system board, different mezz adapter card family and it has to talk to all those server devices which don't exist on ProLiant.. ..so there's Synergy specific features that must work correctly.   We are working to be more diligent to remove statement of support for Synergy when that validation has not yet occurred (and either the iLO support pages will change or the next version that works for all adapters, memory, and CPU options will get released and supercede it.

To deal with Intel, back in 2018(?) we allowed immediate update of BIOS ROMs.  So the facts are that the HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur.  Anything else is risky.  We were working to incorporate flexbility a few components at a time but then we hit issues with precisely the two components we wanted to allow open updates for.  So lesson learned, just in time you could say.

Longer term, all this will change late this year.  Stay tuned and please realize the valiant start would have been retired the instant it was no longer necessary but that time has not arrived.  Even with our CNAs you'll often find that updated driver/firmware will work for you until you try to use something beyond basic networking (such as FCoE or VPLAG/downlink LAGs, etc.).  When we publish support, it must be for the entire feature breadth across all currently supported options and even if 2/3 our customer base isn't using what's broken we can't let that go out.

Edit:  

• Customer Notification a00050817en_us.

[PatrickLong]

@DanCerneseThank you very much for this clear, concise, reasoned explanation.  I'll take your line "HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur. " as gospel going forward for my Synergy environments, which (other than the extraneous iLo update) are all set at the latest Synergy Custom SPP 2019.12.20200125 .  In the meantime, I'll set about the task of back-rev'ing the iLo's on those systems from 2.12 to 2.11.

Cheers, Patrick

[PatrickLong]

@DanCernese  I see a new standalone download Gen10 iLo5 release v2.15 dated April 9th, 2020.   Interestingly, two of the five listed Fixes in this version specifically mention Synergy:

-The HPE Synergy 3820C 10/20Gb CNA loses flexed sub-ports after a cold boot or E-fuse reset.
-iLO is temporarily unresponsive or iLO resets on HPE Gen10 servers and HPE Synergy Gen10 compute modules running HPE Infosight.

and one of the three Enhancements is specifically for Synergy:
-Users can now restore the last power state after a power outage on Synergy compute modules.

I'm honestly not trying to be provocative, but is your guidance still "the HPE Synergy Custom SPP is still the only supported receipe-- plus BIOS ROM updates as they occur", or is iLo now moved a relaxed cadence like the BIOS?  You did mention in your last response that "Longer term, all this will change late this year." but I'm not sure if this is what you meant by that.  And if your guidance remains the same, can I infer that we are very close to a holistic Synergy Custom SPP for April/May 2020 that will include this new 2.15 version for iLo5 - and it just hasn't made its way through the publishing pipeline just yet?  My Synergy node iLo5's are feeling a bit long in the tooth at version 2.11 dated Nov 12 2019 (which is the latest available on the latest Synergy Custom SPP Version 2019.12.20200326) when my rack-mounted Gen10's get to use a fully-supported 5-months and three-releases-newer v2.15

[DanCernese]

Hello @PatrickLong ,

Yes those are Synergy fixes and they are incorporated into the soon to be HPE Synergy Custom SPP 2020.05.01 awaiting web publication.  The backend IT and supply chain publication is in process to release that to the web this month.  That will, of course, bring with it the whole firmware baseline and OneView 5.2.

We have requests from customers in the same situation who want those same iLO5 fixes without requiring update to their existing infrastructure.  This is a serious and well-recognized request in product management.  However, the newest iLO5 also drags along with it a requirement to use a fixed HPE SUM, and therefore the validation process becomes complex (read: expensive).  Suffice to say we are evaluating if and how we might release the iLO5 update either by itself or against existing infrastructure.. ..but today that's only a discussion.  

[PatrickLong]

It looks like the HPE Synergy Custom SPP 2020.05.01 has now been published with the release of OneView Composer 5.20.00

As indicated, iLo5 21.5 is included in that new custom SPP along with a number of other updates. 

Thank you @DanCernese