Error : "Problem manipulating EV" and Status: 51

[meant0m]

We are experiencing an issue when assigning new profiles to our  SY 480 Gen10 CMs where we end up receiving the following error during profile assignment:

"Error : "Problem manipulating EV" and Status: 51"

The CM will just get stuck in a boot loop where it will reach the end of POST and then shows the text that it needs to reboot to apply the changes.  We have opened 5 cases for this with HPE and they just replace the system board.  The horrible part is that we have seen about 25% failure rate so we lose a server completely until they replace the system board.

Why is this happening and is there something we can do differently to have better success?  We gracefully shut down the CM, remove the current profile, then add the new profile.

We are running the following firmware versions:

HPE Synergy Service Pack, SY-2022.11.01

OneView v8.00.00-0470555

[DanCernese]

I saw traffic on this recently too.  I believe there is a discovered issue with a fix coming for Synergy but not yet released.  I encourage you to pursue a support case with a support dump and AHS log.  I am not personally aware of workaround or recovery offhand I saw in one case that resetting the BIOS to factory defaults cleared the condition (but no promises).

[meant0m]

Hi Dan,

Yeah, we have tried resetting to system defaults and factory defaults from BIOS.   Tried reseating the CM pre and post-profile applications to try to clear it out completely.  

Unfortunately, we are in the process right now of replacing the "Synergy 3530C 16G HBA" card with the "Synergy 5330C 32Gb FC HBA" on 48 compute modules and have lost a couple of servers already because we have to apply a new (but very similar) profile because Synergy recognizes the Compute Module as a different server hardware type.

[ChrisLynch]

You are changing the server hardware within, which will create a new server hardware type. The SHT describes the capabilities and installed options that OneView can manage. Which really mean the adapters. At a minimum you need to change the Server Hardware Type in the profile so it can be re-applied. At most you create a new server profile template then create a new profile.

You can always create a profile template from your original profiles or copy an existing one. Then change the server hardware type there to then create server profiles.

[meant0m]

Thanks, Chris.  We are familiar with the methods of either using the same profile or creating a new one.  Regardless, the issue still persists in which the profile application fails with the error "Error : "Problem manipulating EV" and Status: 51".  This doesn't appear to happen with all compute modules, just about 25% of them.

[meant0m]

UPDATE!  

Found this article: https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00126447en_us

and updated the ilo to 2.78 and performed the remaining steps listed in the article. We were able to successfully apply the profile without receiving the error.  We are going to try on more hosts today.

[DanCernese]

I am aware of a fix in that version, but I did not specifically mention it because it is NOT SUPPORTED ON SYNERGY.

You will hit other issues with that version.  The version that carries the fix you require along with passing all the Synergy validations should be available in a few weeks.

[meant0m]

Hi Dan,

I am a little confused because I was looking at the release notes for Composer2 OneView 8.20 thinking maybe the fix was included in that release. (https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_2a9e050c12944cd3b7b9e54d05&tab=Fixes)  and you can see in that article under the section titled "HPE Synergy Service Pack (SSP) 2022.11.01 - November 2022" in which it says,

"Resolved – On an HPE Gen10 or HPE Gen10 Plus server with HPE Integrated Lights-Out 5 (iLO 5) firmware version, server profile creation in HPE OneView may intermittently fail and display the following message: ;Unable to apply server profile settings to the management processor;. For more information, see Customer Advisory a00126447."

The link they show for the Customer Advisory was what I shared in my previous post so to me it seemed like it would have applied to Synergy.

[ChrisLynch]

Thanks, Chris.  We are familiar with the methods of either using the same profile or creating a new one.  Regardless, the issue still persists in which the profile application fails with the error "Error : "Problem manipulating EV" and Status: 51".  This doesn't appear to happen with all compute modules, just about 25% of them.

I wasn't offering a solutoin tot he EV issue.  Dan has been helping you with that already.  I was simply providing more explination as to why the server hardware type would change and how to handle those changes with profiles and profile templates.

[meant0m]

The information was much appreciated Chris and thank you for your prompt response.  

[meant0m]

@DanCernese 

Dan,

What is also confusing is that we received a Security Bulletin that identifies a vulnerability that is resolved by applying ilo 5 v2.78 for Synergy 480 Gen10 so we felt like it would be safe to upgrade.  We already have applied the 2.78 ilo firmware to 32 of our 48 CMs.

The Bulletin I am referencing:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-hpesbhf04456en_us&hprpt_id=ALERT_HPE_3043901&jumpid=em_pom8nu6hj_aid-520066529

[DanCernese]

Yes, I understand.  I apologize forr the confusing communications from HPE.  This was the plan but it changed and the security bulletin was already in flight.  Be aware that iLO 5 2.78 may demonstrate issues and if you have the opportunity, please evaluate the next firmware bundle if it matches the rest of your infrastructure and computes (i.e. supported OneView and OS versions).