Firmware baseline not updating all components

[Kerry Quillen]

I have created and uploaded a custom SPP (using the 2019.03 SPP) for my Gen10 DL and BL managed servers in Oneview (v4.20.02).  This of course generates "profile is inconsistent" alerts.  After I execute "update from template" the DL servers will return an alert stating "One or more firmware components are inconsistent with the specified firmware baseline".  Turns out that all firmware components were upgraded except the Smart Array P408i-a SR Gen10 Embedded RAID controller.  If I mount the same ISO with ILO virtual media and boot from it, it upgrades all firmware including the array controller.  As a test, I uploaded the the full 2019.03 SPP and applied that via the Oneview profile with the same result - RAID controller does not upgrade.  Tried reapplying the profile a second time but still no luck.  I can't find anything within Oneview to indicate why that one component gets skipped.    I do firmware only updates in my profiles. These are all VM host servers.

[MV3]

HI kerry,

Good Day!

Can you please try Power cycle the server and wait for the POST to complete, and reapply the server profile to update the firmware again and check.

Cheers,

[Kerry Quillen]

I tried that.  Didn't work.  I created and uploaded a new custom SPP based on the latest 2019.09 SPP.  Applied that to a test server via the template/profile and it worked.   I'll roll that out during our next firmware update cycle which will be 6-9 months.

[Kerry Quillen]

I went ahead and created and tested a new template on 2 servers with the 2019.09 spp and it did the same thing.  I thn tested a 3rd by mounting the spp iso through virtual media and it worked.  I did however notice that instead of making one pass through it made 2.  The 2nd time around I saw this.  I can't find cp039732.zip in the spp so have no idea what it is but assume is the array controller firmware

[Kerry Quillen]

I'm still fighting this issue.  Anyone else seeing this problem?  I have multiple instances of Oneview and I get the same results in each using the same custom spp iso.    

[BrettPowell]

Did you ever resolve this issue? I am dealing with the exact same issue with the Service Pack for ProLiant version 2019.12.0. Moderator [above link is no longer valid, please visit https://support.hpe.com/connect/s/  to find the latest info ]

I can update the firmware by mounting the .iso manually and updating it but it won't do it through the profile with the same exact baseline.

[Kerry Quillen]

No I haven't.  In fact my next step was to try the 2019.12.0 SPP but looks like you saved me the trouble.  Guess I'll have to open a support case.

[Kerry Quillen]

Well here you go - https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00096139en_us

[Kerry Quillen]

So the workaround for that bulletin says:

1. Delete all Service Pack for ProLiant (SPP) releases from HPE OneView.
2. Add back the latest SPP first.
3. Add the other SPPs and run the server profile apply operation using any of the SPPs.

If I do that is it not going to make all of my profiles inconsistent and require them to be updated again?  I just finished a round of firmware updates on all of my servers.  Don't really want to go through that again right now. 

[Anonymous]

Yep (same problem).

The December update was fine.

Latest OneView (5.2) and latest SPP (May 2020) blade updates fail when applied via profile.

Case opened, HP want logs (not going to happen). 

Will advise any resolution. 

[Anonymous]

Tried removing and re-adding the latest SPP before reapplying - same result.

Haven't tried deleting *all* and re-adding latest yet; that may be the next step...

[Kerry Quillen]

I deleted all of my spp bundles and added them back, starting with the newest first.  Still fails.

[Kerry Quillen]

Has anyone made any headway with this?  Last HPE support person I spoke with (it was a different issue) said updating to 5.20 should help.  It did not.  I'm now using custom spp built off of 2020.03 spp.

[BSTRD]

I have the same issue on Oneview 5.3 and SPP 2020.03 + June supplements + hotfixes

I have a case that was escalated to L2 two weeks ago without making any real progress. They are advising the same thing with removing and adding all SPP's which

1) will mean maintenance reboots for all servers (unacceptable)

2) the CA is referering to only SPP2019.12 as affected but the current builds are +6months older and if pointing this out then radiosilence ensues from support.

Is there a grown up person from HPE that is monitoring this thread and can step up?

/Morten

[ChrisLynch]

@BSTRD, sorry to hear you have this issue. Please know that setting the baseline to Manage Manually does not require the server to reboot, nor power off. Unless there is a other setting within the profile that has yet to be deployed correctly.

Can you private message me your support case, and I will have someone help move this along.

[BSTRD]

So my case is still ongoing and progress with L2/3 so far is that they are unable to reproduce the issue and still have not given a suggestion of something to try out in the customers environment to try an fix the issue.

Also have another case open from the same customer where Oneview is unable to apply updates in online mode using SUT but SUM on the same SPP does not have problem when running it interactively on the server.

Very unhappy!!

[TobiasS]

Hello together,

on our side, we have the same problem with OneView 5.40.00-0423953 and the newest SPP2020.09 on  DL Gen10 servers too.

Last time I added a new SPP, I had this problem and could fix it be removing and adding the SPP's again, like others described here and also in this Advisory: https://support.hpe.com/hpesc/public/docDisplay?docId=a00096139en_us&docLocale=en_US

But I don't want to go this way again, especially since some people have already written here that it didn't help them.

DL G9 works fine, but with iLO4 it is a completely different situation.

Please keep us informed if there are some new information.  In my opinion, the problem has existed for too long and make a lot of trouble for importing server systems.

Regards Tobi

[BSTRD]

Hi

In the end then ERT (L3/L4) support acknowledged that this issue would not be fixed until SPP 202009 (with SUM 8.7) was released - the knowledge base article was since updated to reflect this statement, however it is still not accurate as this customer had the problem also on Oneview 5.1 - 5.4, the error message changed slightly after update to 5.3 but the core issue remained.

The core of the problem is that the implementation of the SUM package parser (prior to SUM 8.7) is flawed when used inside Oneview.

The issue is the same regardless of doing online or offline application of firmware baselines in Oneview.

If you use the embedded version of SUM to update the servers directly (outside Oneview) then it works fine. Still ERT support was adamant that the problem resided with SUM and not with Oneview, so I am unsure if the version of Oneview matters.

The problem also manifested itself regardless of using standard or custom version SPP's, both with and without additional hotfixes added to the baselines inside Oneview - all combos was attempted.

There was an additional attempt with a scripted solution resetting the SUM cache in Oneview (this is only avail via support as it requires support login on the appliance). This forces Oneview to rebuild the baseline from scratch and is less impact and less work than what is suggested in the article with unassigning and removal of all SPP's.

In the end (since L3) was unable to reproduce the problem and couldnt provide a solution, the client opted to wait for SPP 202009 and install a new clean Oneview 5.4 appliance and only upload SP 202009 to this appliance and migrate their servers to this baseline - this has worked fine (so far - migration is still in progress) on both Gen10 and Gen9 servers.

For customers still having this issue then before migrating to a new appliance, I would try the suggestion from the article of removing all SPP's but also open a support case and do a SUM cache reset + only adding SPP 202009 after the reset and use only this SPP for all servers. If still having Gen8 servers then manage those servers on a separate instance and monitor using the global dashboard.

/Morten

[TobiasS]

Hi Morten,

thank you very much for your quick answer.

I removed all SPP's and also all "managed" Gen8 Servers (re-added as "monitored" only).

After this I added the 2019.9 SPP again and tried to apply the profile changes to Gen10 ESXi 6.7 Hosts.  This time it looks much better, so I hope it will stay stable. I didn't open a case, so no SUM Cache reset was done.               

If the problem will come back by adding a new SPP in the near future, I will do the same, just with the SUM Cache reset (Case).

I just have a few Gen10 ESXi Hosts left, where I cannot install some firmware updates. Most time the cp043298 failed, also after cold boot. Here are to example messages:

The firmware update operation was unable to update the following components on server ABZ:
cp043298.exe (SHFW_036)

The firmware update operation was unable to update the following components on server XYZ:
cp043298.exe
cp043596.zip
cp040927.zip (SHFW_036)

Any idea how I can fix this?

Stay healty

Tobi

[BSTRD]

Hi

Yes, I also see some failures on different packages sometimes, but at least now an attempt to install them is made - significant progress from earlier where it didn't even get to that point. I have not been in a place where I could not fix it manually by either interacting with the server directly (during profile application) or do a re-apply of the profile.

I monitor the ilo firmware queue to see if the package gets placed there correctly - after that the application of the packages happens more or less autonomously on the server - Oneview seems to progress normally even if the server needs a "jumpstart" to proceed correctly sometimes. Have also seem the server stopping during the remote iso boot to SUM and then applied the ilo firmware queue manually by clicking in the console gui.

As for adding new SPP then recently ilo 2.31 was released and I needed to add that to a new baseline inside Oneview (fixes the false positive error with ram sensors in vmware - https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00106468en_us) .

I was necessary to add both the windows, linux and standalone package version to Oneview before the hot fix got applied to the server - and SUM stages both the 2.30 + 2.31 versions to the server and also applies both packages on the server - so they are not done looking at the SUM parser and none of that info seems to be in any documentation that I could find.

Oh and yes, there has been complete radio-silence from all the people on my mailing list on the previously mentioned support-case when I sent in my observations.

/Morten