CPLD firmware update for SY 660?

[PatrickLong]

I note that there is a new version ( 0x0F ) of Complex Programmable Logic Device firmware released for Synergy SY 480 Gen10 and Synergy SY 48- Gen10 Plus (Note there is different file for each!) in .fwpkg format on the Support Site,  BUT I am unable to find any similar new releases for SY 660 Gen 10 or the SY 660 Gen10 Plus .

The latest I am able to find for the SY 660 Gen 10 is 0x1C  from 2019 linked in these two older Customer Advisory documents:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00076367en_us and

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00077281en_us

I personally have a number of SY 660's suffering the amber light cosmetic issue described above that are resolved in newer CPLD firmware releases, and somehow missed the advisory in the past - but I can't find any links on the support site, and the links in the advisories above are for Win and Linux binaries. My hosts are VMware ESXi.  Why are there no new CPLD releases for SY 660 G10 and G10Plus in the .fwpkg format that I can apply via iLoREST as there are for the SY 480?

Or perhaps even more importantly, WHY HAVE FIRMWARE UPDATES FOR THE CPLD NEVER BEEN INCLUDED on any version of the older Synergy Custom Service Pack for ProLiant SPP OR the current Synergy Service Pack SSP iso's?  This is just like the Innovation Engine and Server Platform Services components, which in the early Synergy 2017-2019 days were never included in early SPP releases and required a separate convoluted (read: not documented) install process to load them on Synergy Compute modules running ESXi - these two components were only relatively recently added to the SPP/SSP.  I mean, the whole point of the SPP and SSP .iso' s is to provide an HPE- blessed recipe of firmware and driver versions that are tested to work together on Synergy so that I'm NOT downloading one-off component firmware updates individually. It should include CPLD firmware imo.

If anyone can point me to an SY 660 Gen10 CPLD firmware NEWER than the 0x16 or 0x17 that I find on many of my SY 660's - and newer than the 0x1C version from 2019 Advisory linked above - preferably in .fwpkg format, I would appreciate it.  I have opened a Support case for this as well.

[support_s]

System recommended content:

1. Updating firmware

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[ChrisLynch]

Or perhaps even more importantly, WHY HAVE FIRMWARE UPDATES FOR THE CPLD NEVER BEEN INCLUDED on any version of the older Synergy Custom Service Pack for ProLiant SPP OR the current Synergy Service Pack SSP iso's?  

CLPD was never intended to be updated in the field, per the conversations from the server hardware teams diagnosing this.  In this case affecting Synergy Gen10/Gen10+ servers, the CPLD firmware needs to be updated to address the underlying condition.  There has never been a need prior.

As for when a SY660 Gen10/Gen10+ CLPD firmware update, I am looking into it, as I thought it was supposed to be released as well.

[PatrickLong]

Thank you for this info, Chris.  As I look across my smallish (38) fleet of SY660's, I see CPLD versions including:

0x14

0x16

0x17

0x1C (on a single server I just upgraded by downloading the Windows package linked in the 7/01/2019 Customer Advisory a00076367en_us and also the 7/10/2019 Customer Advisory a00077281en_us, then extracting the CPLD_SY660_Gen10_V1C1C_signed.vme file and uploading it into the iLo5 >Update Firmware function as directed by Support. I'm a bit surprised I missed that Advisory as I definitely have SY660's that are afflicted with the cosmetic-only errant amber light on the mobo.

<If> there is also a new 0x1F version that was supposed to have been released recently with the corresponding SY480 files as you (and I) thought, that would make at least five different releases of the SY660 CPLD firmware to date, which seems like enough revisions to at least consider for inclusion in future SSP releases.  I admit, it is a bit more invasive an update than other firmware components as it's very low level and initiates an auxiliary power cycle (Device in Bay xx has been removed) on restart - so that's a consideration as well.

In any case, thank you for responding and please do post here if you are able to find a newer release than 0x1C.

Patrick

[PatrickLong]

Slightly off-topic, but I now see that an Advisory a00121027en_us has been released for the SY480 related to new SY480 CPLD firmware was published yesterday, which raises it's own questions.  The Advisory does very specifically refer to a precise serial number range to which the fault condition applies which REQUIRE the CPLD firmware as the fix for this fault.  But I wonder if there is a compelling reason why I should simply NOT update ALL of my SY480's (all of which are at CPLD 0x0A, and none of which are in the Advisory serial # range) to this newer CPLD firmware 0x0F, so long as I am aware of and accept the relative invasiveness of updating that component,  with it's auxiliary power cycle requirement, etc. as outlined in the Advisory.  What is the down side of doing that?

As you may have guessed, I am not an "if it ain't broke don't fix it" guy; I'm more of a "newer versions provide compatibility and algorithmic efficiency improvements IN ADDITION TO bug fixes - even to bugs you haven't personeally experienced yet, but you might someday if you leave it at this old version" guy.  I'd rather "fix" things in my environment now at my own pedestrian pace and cadence - first in a test/lab setting and then after some time bring it to production - than wait for a catastrophic issue to bite me and have to scramble to remediate it on someone else's schedule, only to find out that the fix was a 2 year old firmware that I never applied because it wasn't broken at the time.

[ChrisLynch]

If there is also a new 0x1F version that was supposed to have been released recently with the corresponding SY480 files as you (and I) thought, that would make at least five different releases of the SY660 CPLD firmware to date, which seems like enough revisions to at least consider for inclusion in future SSP releases. 

There are no plans to include this as part of an SPP going forward.  It is an extremely low-level component that to this day has only been updated in the factory.  There has been no need to release CLPD firmware to the field, except for this specific event.

The Advisory does very specifically refer to a precise serial number range to which the fault condition applies which REQUIRE the CPLD firmware as the fix for this fault.  But I wonder if there is a compelling reason why I should simply NOT update ALL of my SY480's (all of which are at CPLD 0x0A, and none of which are in the Advisory serial # range) to this newer CPLD firmware 0x0F, so long as I am aware of and accept the relative invasiveness of updating that component,  with it's auxiliary power cycle requirement, etc. as outlined in the Advisory.  What is the down side of doing that?

No.  Please follow the guidance documented in the advisory.

As for the SY660 Gen10/Gen10+ CLPD firmware, it hasn't been released yet.  I haven't been able to find any ETA right now, but will continue to research this.

[Lawrance Lee_1]

Is it possible to apply the CPLD firmware en-masse to a fleet of servers, and then reboot the servers at a later stage, when it's possible to arrange a maintenance outage?  In other words, is it like an online BIOS update where the new version remains dormant until such time the server is rebooted, and the new version comes into effect.

[PatrickLong]

@Lawrance Lee_1  I don't believe it works like that, as all of the Advisories I can find for CPLD updates on Synergy compute modules - one for the SY660 Gen10,  one for the SY480 Gen10 and one for the SY480 Gen10 Plus all state some variation of the instructions  that the server should be powered on  prior to the CPLD flash, and "some variation of "After the flash is completed, reboot the server."   That leads me to believe that CPLD is not a redundant ROM sutiation like you have with BIOS - there is likely only one CPLD component on the motherboard and you should reboot immediately after completing the installation per those Advisories. 

[PatrickLong]

@ChrisLynch Per my HPE Support Case they indicated that there is NOT a new version of CPLD firmware for the SY660 and that the 0x1C version indicated in the 2019 Advisory is the latest - so I'll take that as the answer unless you dig up different information.

However, in response to your answer that I should NOT update any of my SY480 Gen10's (all of which fall outside of the SY480 Gen10  Advisory's scope serial number range) I would respectfully ask that HPE add that important information somewhere on the Support Site download page for CPLD_SY480_Gen10_v0F0F.fwpkg.  There is nothing anywhere on the download page tabs indicating a restriction by serial number range for which SY480 Gen10 servers it should be installed.  If I never see the Advisory linked above but just go to the Support Site as part of my regular administratoive due diligence and search on SY480 Drivers and Software filter to see what's new, I'll find that CPLD update rated as as an Upgrade Requirement: Critical, yet never know to only apply that to SY480 Gen10's of a specific serial number range. Reconciling the Advisory info to the download page info would be helpful, or incluing a link to the Advisory itself on the Prerequisites tab of the download would also be helpful.

[ChrisLynch]

I don't know who provided that answer about SY660 to you, but that isn't what the Synergy team has been working on nor what was told to me the other day.
As for the advisory, I'll pass that along.

[DanCernese]

"...there is NOT a new version of CPLD firmware for the SY660..."

Yet.  

For the SY660 S/N, the issue is only in the range of 116 through 210.  to be published as soon as practical, but the logic is not the same and triage and validation cycles are still ongoing.  no forecast at this time.

[PatrickLong]

@DanCernese  So if I am understanding HPE recommendations correctly...  I should apply the 0x1c SY660 CPLD update from 2019 per Customer Advisory a00077281en_us to resolve the stuck-on-amber inconsequential mobo light on my SY 660's - regardless of their serial # designation - BUT I would only need to then subsequently apply this yet-to-be-released SY660 CPLD to SY660's in the serial # range 116 through 210 as the 4th through 6th digit per guidance.  None of my SY660's fall into that serial range, so I will NOT need to apply the new CPLD when it is released.  Do I have that correct?

Extra credit Dev assignment:  Can HPE please program logic into the CPLD binary that reads in the current mobo serial# and evaulates via regex to determine if it should be applied or not?  Particularly if it is strongly not recommended (or even potentially dangerous) to apply the forthcoming CPLD to an SY660 with a non-qualifying serial #... relying on human eyes to make this assessment is fraught with risk.

[PatrickLong]

I see that the CPLD firmware 0x1D for a subset of SY660 comute modules as well as a Customer Advisory a00121347en_us has now been released, and as with the SY480 CPLD release, this Advisory identifies very strict serial number range requirements for proper application of this CPLD firmware in the Advisory document's Scope section.  However, also like the SY48 release, this same serial number range restriction information is NOT included on either the Installation Instructions or the Release Notes tabs of the actual download page - this is an inconsistency that is very likely to cause confusion and misapplied updates along with whatever potential negative ramifications that would bring.  If this scope restriction to only a subset of serial numbers is in fact as important a requirement for the CPLD releases as the Customer Advisories for both the SY480 and SY660 would lead one to believe, I would ask that HPE please update those download pages with the same information for the sake of consistency.

[DanCernese]

fyi-- SY660 have 2 CPLD. Engineering discovered that early models of SY660 that were outside the affected SN range have a very early version of CPLD on one of the chips.. ..and that's the one that fails to flash correctly from that old version to the newest.  So we will emphasize NOT to apply outside the SN range for SY660.  In parallel we're researching if we can modify the package to check first; but that turns out not to be trivial for CPLD.

[PatrickLong]

@DanCernese any further info on this SY660 CPLD issue that you can divulge here?  I have an SY660 that is well outside of the S/N range 116 through 210 specified in the Advisory, (my 4th-6th digits are 842) yet this server has now failed twice in production with one of the failure modes listed in the Advisory "the system may power off abruptly during runtime and require an E-fuse reset to restart it."  Also same as the SY660 CPLD 0x1D download page Description tab  "1. A runtime shutdown with no standby power and no associated entry in the IML." which required an e-fuse reset to power on the server. Absolutely nothing identifiable in IML,  AHS, or ESXi  logs related to the source of these failures.

I had opened Support Case 5365360842 where they eventually tacitly advised me to upgrade the CPLD from current 0x1C to 0x1D:;  Support stated  "We do not see Fault Start / Fault Stop in AHS all event logs, however we have heard that there are servers which are out of that range which could possibly be affected.  If the system is up now, then go ahead and flash the CPLD FW.".  I'm just looking for your thoughts before I move forward and flash this thing, particularly given the May 9th, 22 addition to the Advisory "IMPORTANT: The CPLD version 0x1D must not be applied to HPE SY660 compute modules which are not in the affected serial number range. Doing so may prevent the compute module from powering on and require downgrading to an older CPLD version in order to recover." 

[DanCernese]

I am not aware of engineering having worked on this topic any further and I believe taking action not based on data from engineering is very risky..  I am unaware of any server outside the range being affected of any model unless the system board had been replaced.

[DanCernese]

I see they closed the case; please ask them to re-open it and elevate to engineering and point them to the advisory IMPORTANT note.  Given that the symptoms do not match, I would not apply the update unless you are prepared to potentially have to recover the server in the case where it does not resolve the issue.