Firmware baseline not updating all components

[Anonymous]

Tried removing and re-adding the latest SPP before reapplying - same result.

Haven't tried deleting *all* and re-adding latest yet; that may be the next step...

[Kerry Quillen]

I deleted all of my spp bundles and added them back, starting with the newest first.  Still fails.

[Kerry Quillen]

Has anyone made any headway with this?  Last HPE support person I spoke with (it was a different issue) said updating to 5.20 should help.  It did not.  I'm now using custom spp built off of 2020.03 spp.

[BSTRD]

I have the same issue on Oneview 5.3 and SPP 2020.03 + June supplements + hotfixes

I have a case that was escalated to L2 two weeks ago without making any real progress. They are advising the same thing with removing and adding all SPP's which

1) will mean maintenance reboots for all servers (unacceptable)

2) the CA is referering to only SPP2019.12 as affected but the current builds are +6months older and if pointing this out then radiosilence ensues from support.

Is there a grown up person from HPE that is monitoring this thread and can step up?

/Morten

[ChrisLynch]

@BSTRD, sorry to hear you have this issue. Please know that setting the baseline to Manage Manually does not require the server to reboot, nor power off. Unless there is a other setting within the profile that has yet to be deployed correctly.

Can you private message me your support case, and I will have someone help move this along.

[BSTRD]

So my case is still ongoing and progress with L2/3 so far is that they are unable to reproduce the issue and still have not given a suggestion of something to try out in the customers environment to try an fix the issue.

Also have another case open from the same customer where Oneview is unable to apply updates in online mode using SUT but SUM on the same SPP does not have problem when running it interactively on the server.

Very unhappy!!

[TobiasS]

Hello together,

on our side, we have the same problem with OneView 5.40.00-0423953 and the newest SPP2020.09 on  DL Gen10 servers too.

Last time I added a new SPP, I had this problem and could fix it be removing and adding the SPP's again, like others described here and also in this Advisory: https://support.hpe.com/hpesc/public/docDisplay?docId=a00096139en_us&docLocale=en_US

But I don't want to go this way again, especially since some people have already written here that it didn't help them.

DL G9 works fine, but with iLO4 it is a completely different situation.

Please keep us informed if there are some new information.  In my opinion, the problem has existed for too long and make a lot of trouble for importing server systems.

Regards Tobi

[BSTRD]

Hi

In the end then ERT (L3/L4) support acknowledged that this issue would not be fixed until SPP 202009 (with SUM 8.7) was released - the knowledge base article was since updated to reflect this statement, however it is still not accurate as this customer had the problem also on Oneview 5.1 - 5.4, the error message changed slightly after update to 5.3 but the core issue remained.

The core of the problem is that the implementation of the SUM package parser (prior to SUM 8.7) is flawed when used inside Oneview.

The issue is the same regardless of doing online or offline application of firmware baselines in Oneview.

If you use the embedded version of SUM to update the servers directly (outside Oneview) then it works fine. Still ERT support was adamant that the problem resided with SUM and not with Oneview, so I am unsure if the version of Oneview matters.

The problem also manifested itself regardless of using standard or custom version SPP's, both with and without additional hotfixes added to the baselines inside Oneview - all combos was attempted.

There was an additional attempt with a scripted solution resetting the SUM cache in Oneview (this is only avail via support as it requires support login on the appliance). This forces Oneview to rebuild the baseline from scratch and is less impact and less work than what is suggested in the article with unassigning and removal of all SPP's.

In the end (since L3) was unable to reproduce the problem and couldnt provide a solution, the client opted to wait for SPP 202009 and install a new clean Oneview 5.4 appliance and only upload SP 202009 to this appliance and migrate their servers to this baseline - this has worked fine (so far - migration is still in progress) on both Gen10 and Gen9 servers.

For customers still having this issue then before migrating to a new appliance, I would try the suggestion from the article of removing all SPP's but also open a support case and do a SUM cache reset + only adding SPP 202009 after the reset and use only this SPP for all servers. If still having Gen8 servers then manage those servers on a separate instance and monitor using the global dashboard.

/Morten

[TobiasS]

Hi Morten,

thank you very much for your quick answer.

I removed all SPP's and also all "managed" Gen8 Servers (re-added as "monitored" only).

After this I added the 2019.9 SPP again and tried to apply the profile changes to Gen10 ESXi 6.7 Hosts.  This time it looks much better, so I hope it will stay stable. I didn't open a case, so no SUM Cache reset was done.               

If the problem will come back by adding a new SPP in the near future, I will do the same, just with the SUM Cache reset (Case).

I just have a few Gen10 ESXi Hosts left, where I cannot install some firmware updates. Most time the cp043298 failed, also after cold boot. Here are to example messages:

The firmware update operation was unable to update the following components on server ABZ:
cp043298.exe (SHFW_036)

The firmware update operation was unable to update the following components on server XYZ:
cp043298.exe
cp043596.zip
cp040927.zip (SHFW_036)

Any idea how I can fix this?

Stay healty

Tobi

[BSTRD]

Hi

Yes, I also see some failures on different packages sometimes, but at least now an attempt to install them is made - significant progress from earlier where it didn't even get to that point. I have not been in a place where I could not fix it manually by either interacting with the server directly (during profile application) or do a re-apply of the profile.

I monitor the ilo firmware queue to see if the package gets placed there correctly - after that the application of the packages happens more or less autonomously on the server - Oneview seems to progress normally even if the server needs a "jumpstart" to proceed correctly sometimes. Have also seem the server stopping during the remote iso boot to SUM and then applied the ilo firmware queue manually by clicking in the console gui.

As for adding new SPP then recently ilo 2.31 was released and I needed to add that to a new baseline inside Oneview (fixes the false positive error with ram sensors in vmware - https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00106468en_us) .

I was necessary to add both the windows, linux and standalone package version to Oneview before the hot fix got applied to the server - and SUM stages both the 2.30 + 2.31 versions to the server and also applies both packages on the server - so they are not done looking at the SUM parser and none of that info seems to be in any documentation that I could find.

Oh and yes, there has been complete radio-silence from all the people on my mailing list on the previously mentioned support-case when I sent in my observations.

/Morten