Getting sick of this now...: Power Management Controller Firmware

[T_1_6]

Tried to update our DL360 Gen10 estate with the 2023.09.00.00 SPP. Big mistake, literally NOTHING WORKS.

Vmware lifecycle manager fails with this, then you go through all the hoops to do it manually via Oneview, and it is stuck and NOTHING, I repeat, NOTHING works to get past this error/inconsistency with updating the Power Management Controller firmware from 1.0.8 to 1.1.0.

Sorry HPE, but you have driven me to the edge now, I never want to use your server products ever again. Vmware Lifecyle manager has NEVER worked, ever, not properly or seamlessly. Firmware updates are a total lottery, and now we have  a load of esxi nodes stuck with this firmware issue where it simply refuses to update.

I can only assume this is a bug with the 2023.09.00.00 SPP? 

I gotta be honest, I am sick of days and days of my time being wasted to do such a simple thing as a firmware upgrade.

No clue how this can be solved... we are stuck with a load of inconsistent baselines now, and they simply refuse to update the power controller firmware, whatever it is, I dont care, I dont need to know, I dont want to know, I just want something to actually work for once. Just once, thats all.

Anyone got any ideas?

The following firmware components are inconsistent with the specified firmware baseline: Power Management Controller Firmware

[T_1_6]

Looks like the update took out our SmartArray Controller, we have no boot drives any more.

Great stuff.

[CarlPowerNZ]

I have about 320 hosts to update with the new SPP, and the PMC also is not updating, but everything else is updating.

I do not use the SPP in vLCM at present, as we have an open case with HPE and VMware, where the SPP invalidates some factory VMware drivers, and removes access to the latest driver versions, so setting up vLCM clusters is near impossible - if you want some help with setting up vLCM for your clusters, heres what I did with lots of help from the Denver VMware support guys:

1. If you have registered any SPPs in your VMware LCM, you will likely need to reset the VMware update DB to recover factory driver versions - skipping this may cause you immense pain!  Check if there are any registered - if so, you will need to reset the LCM DB
   
   
   
   
2. For any vSAN witness hosts, reverting it to a vanilla ESXi instances will save lots of pain - you need to SSH to the witness and reset the profile:

For build 21930508, where /locker/packages/ contains a copy of the ESXi image.

esxcli software profile install -d /locker/packages/VMware-ESXi-7.0U3n-21930508-depot.zip -p ESXi-7.0U3n-21930508-standard --ok-to-remove​

3. Once the above are ok, create the single image with only the ESXi version and the HPE Customization addon, then add any missing drivers - to not add the SPP to the firmware addon.

[CarlPowerNZ]

I will also add that the SPP update process as a bundle is buggy in general from my experience - there are some components like the SPS firmware that really dislikes being updated with other components...

[T_1_6]

Thanks for your post.
Thats really interesting information, and it's "nice" to know I am not alone in these obvervations and experiences.

We were all sold the dream that you could click a few boxes, and watch vLCM go off and update your whole clusters automatically and seamlessly, and for me it has simply never, ever happened. In fact, I would say its made the process longer, as doing it manually, rebooting into the SPP Linux ISO, patching the server, booting, then applying the esxi update yourself via CLI is actually faster than continually watching all the vLCM/SUM failures over and over, and sittign there for hours restarting the remediation until it finally succeedes. 

[Rama2]

SPP 2023.09.00.00 has 2 PIC (Power Management Controller) components:
Version 1.0.8 is  for Intel platforms.
Version 1.1.0 is  for AMD platforms.

OneView does not handle this at present as our awareness was limited to using supported platforms at the SPP level and not at the component level.

This issue will be addressed in future OneView release. An HPE Customer Advisory build is in progress & will be released once it is finalized.

HPE Engineering team has recommended ignoring this compliance message.

[hauwie]

When will this be solved??

[Vinky_99]

@hauwie 

Good day! 

As they have mentioned "This issue will be addressed in future OneView release. "

[yuriy_tabolin]

We have the same problem with DL360 gen10+ servers, OneView 8.4.0 and SPP 2023.09.00.00. It's pretty inconvenient, hopefully it will be solved soon.

[hauwie]

I see there is a 8.60.00 update of HPE OneView, is this problem fixed in this version?

[Lucas0912]

I am wondering if there is a solution already for this, maybe the new OneView 8.6?

[schmu561]

It made no difference for me.

[mdjmdj]

not fixed in 8.60. 

[hauwie]

I gave up, I removed all our HPE appliances like HPE-Amplifier, HPE-Oneview and HPE-Oneview for VMware vCenter.

I'm going to update the firmware and drivers the 'old school' way.

Mount the HPE SPP (09/2023) ISO, start 'launch_sum.bat', add nodes and just update them.

If you want, you can download the 1.1.0 Advanced Power Capping Microcontroller Firmware from here (https://support.hpe.com/connect/s/softwaredetails?softwareId=MTX_02e6d052f1fd4ad7a1619a8e5c&tab=Installation+Instructions) and upload it to ILO and install seperately.

I'm really sick of companies that are saying we can update drivers and firmwares through VMware with an appliance but don't have support or have well documentation for it. None of the HPE appliances does the job out of the box.

The problems always exists somewhere in the certificate(s).

[schmu561]

What shocks me is how something so obvious made it past testing.  This isn't rocket science, HPE, get your act together.  And this shouldn't take a month to fix.  I know you're all "cloud this and cloud that" but not everyone is interested in getting nickeled and dimed to death with subscription services.

I am thinking I'm just going to start scripting everything and control it that way.  At least you have some control of the quality of the outcome that way.

[pirx]

This thread summarizes everything I've been going through the past 12 months with trying to get vLCM + OV running in a proper way. This was a complete waste of time. And now HPE has the audacity to make OV or Ops manager licenses mandatory if you want to order servers. 

And don't think HPE is really testing anything. They also claimed (in release notes and in mails) that OV + OV4VC +vLCM can update servers in monitored mode for months, but it simply did not work. Then suddenly OV4VC 11.4 was released and it worked. Case closed. I just got feedback... ups... sorry.... it didn't really work before.

The only proper way to let HPE know that they are messing up and generating overhead- which means raising TCO - is to change vendor. The only thing that is currently keeping my company from switching is HPEs aggressive pricing. 

[pirx]

Patch bundle 2023.09.00.01

Software Details - Patch bundle for Gen10 SPP 2023.09.00.00 | HPE Support

Patch bundle version 2023.09.00.01 is an update over September 2023 Gen10/Gen10 Plus SPP version 2023.09.00.00 and primarily includes following updates:

iLO5 2.98

AMS 3.4.0.

Well, only the real importent things are fixed....

[schmu561]

It would be nice if HPE actually had it setup so you could download it.  I've notice they like to post notifications about updates then it takes a week to actually be able to get to them  It's backwards to me, but so is everything else about HPE's web assets.  HPE?  Why don't you lay off the web people you have now instead of the people in the field who give a crap, then have the new web team get rid of all of the floaty circles, rotating dots, and other stuff you're supposed to stare at while waiting for a page that never loads. in favor of something that actually just works.  And please, fix search.

[oaltuntas]

I must say, we are in the same boat.

Upgraded to OV 8.6
downloaded SPP 09.00.01
Gave up on vLCM integration long time ago - so tried to update as usual via Oneview -> Fimrware Only (without SUT crap) -> PMC incosistency errors in Oneview.

Thanks HP. Really thanks. 

We dont have Dells in our team, is Dell any better ? I'm really sick and tired of this and OV vLCM integration that never ever works.

[schmu561]

I will say that, overall, and only recently (the last 12 months) I have had fairly good luck with simple OV updates.  Not perfect by a long shot.  There is unexplained variation each time I do an update, which isn't optimal, but the updates have been applying., until the stupidity of this latest bug with the PMC.  For some reason my iLO passwords will get corrupted after an update and I can find no pattern to explain this.  It will also notify me profiles are inconstent for settings that haven't changed in 3 years.

What galls me is I had to pay for this abuse.  I have always maintained that OV should have been a free product.  Believe it was primarily created to solve the mess they made early on with the C7000 chassis (I was involved in some C7000 FW management conversations with engineering  before OV came to be) and I got the privilege to then pay more to help me fix an issue I didn't create..

[bklee1401]

I just upgraded to that version today and it doesn't look like it has fixed the issue

[schmu561]

Apparently the only person whoc ould fix this was RIF'd..

[pirx]

SPP 2023.09.00.00 has 2 PIC (Power Management Controller) components:
Version 1.0.8 is  for Intel platforms.
Version 1.1.0 is  for AMD platforms.

OneView does not handle this at present as our awareness was limited to using supported platforms at the SPP level and not at the component level.

This issue will be addressed in future OneView release. An HPE Customer Advisory build is in progress & will be released once it is finalized.

HPE Engineering team has recommended ignoring this compliance message.

@Rama2  @ChrisLynch 

Has there been any update on this? It's now 2 months after release and I'm on latest versions of OV4VC (11.4), OV (8.60.01) and SPP 2023.09.00.01. And vLCM update still wants to udapte Power Management Controller Firmware on Intel server to version 1.1.0. And where can I find the mentioned advisory?

[schmu561]

Get a grip, HPE.  In this era where the IT world is filled with bottle-blonde, online MBAs, your dashboard is king, whether it's meaningful or not.  Green is good, even if the feed behind it is garbage..  I didn't suffer such a fate, and I like my dashboard to be meaningful.  If it's filled with a bunch of servers in a warning state for no reason other than HPE can't figure out how to fix something well, then maybe I ignore a legitimate warning as I tread water in the sea of sensors.  How hard can this be?

[mdjmdj]

I gave up on HPE's ability to fix this and created a custom SPP that excludes the AMD Power Management component.   Finally green across the board. 

Power Management Controller Firmware -- 1.0.8 -- OK (1.0.8)