Win 2022 February patches can make windows not boot - No solution from VMWare - KB90947

[Brian_Galante]

There is an issue with this months MS patches where after a Windows server 2022 VM running on ESX Below v8 with Secure Boot enabled is patched, it will not boot up. Not sure what were supposed to do here, VMWare's stance seems to be to upgrade to VSphere 8... See links below. Need some advise from the HPE folks on this.

My workaround for the short term was to disable secure bot on the VM's affected.

See "Issues known in this update section":

https://support.microsoft.com/en-au/topic/february-14-2023-kb5022842-os-build-20348-1547-be155955-29f7-47c4-855c-34bd43895940#:~:text=Known%20issues%20in%20this%20update&text=This%20issue%20is%20caused%20by,for%20Windows%2011%2C%20version%2022H2.

See VMWare response:

https://kb.vmware.com/s/article/90947

[support_s]

System recommended content:

1. HP-UX Consolidated Independent Software Unit (ISU) Release Notes for March 2022

2. HP-UX Consolidated Independent Software Unit (ISU) Release Notes for October 2022

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Mahesh202]

Hi Brian_Galante

I understand that you are facing an issue with Microsoft's recent patches causing Windows Server 2022 virtual machines running on ESX below version 8 with Secure Boot enabled to not boot up.

As for VMware's recommendation to upgrade to vSphere 8, that may be a valid solution, but it may not be practical for everyone. It's always a good idea to keep your environment up to date, but upgrading to a new version of vSphere can be a significant undertaking, requiring careful planning and testing.

Disabling Secure Boot on affected VMs as a temporary workaround is a reasonable solution. However, please note that disabling Secure Boot may decrease the security of your virtual environment.

I recommend reaching out to HPE Simplivity Tech support for assistance with this issue. They may have more specific knowledge of this particular problem and be able to offer more targeted guidance.

They can also provide advice on the best course of action for your specific environment.

In the future, it's always a good idea to test patches in a non-production environment before rolling them out to production to catch any potential issues before they impact critical systems.

Hope this helps.!!

Regards
Mahesh

[ErwinH]

@Mahesh202 

Hi Brian_galante and our HPE friends,

Two days ago the ESXi 7.0U3k has been released (ESXi 7.0 Update 3к | FEB 21 2023 | Build 21313628).
VMware ESXi 7.0 Update 3k Release Notes
In this release the issue with secure boot is fixed! Its now up to HPE Simplivity to aprove this update asap!

HPE friends, can you check and update this post when the ETA of this update is planned?

Regards,
Erwin

[BrianK-54494]

THIS is the fix...  It is imperitive that HPE get this approved ASAP before the security guys strangle us. (  ESXi 7.0U3k has been released (ESXi 7.0 Update 3к | FEB 21 2023 | Build 21313628). VMware ESXi 7.0 Update 3k Release Notes  )

Tempory fix is to disable the secure boot in the VM for patched systems, or to delay patching the VM, but that can only help for a little bit, as it will be in all future cumulative ones I am sure.

[Brian_Galante]

Now we have bots in this forum, thanks for stating the obvious.

[BrianK-54494]

Not a bot, just always been a reader, and never a poster.   I just hope that this gets quicker than normal attention, as our problem with SimpliVity stuff is always that items like this turn into "Well you could probably install it, and don't see anything there that will cause problems, but technically if it isn't on the Interop Guide, you will technically be in an unsupported configuration, blah blah..."     and eventually it shows up on the Interopt or a Paper comes out that says we are good.

This just isn't a security concern that something might happen, it actively is a broke thing (on the second reboot in our testing)... so just hoping it comes quick as possible that we can do the upgrade, as our traditional hosts are all upgrade or last are happening today.

[Kipp_Glover]

Thanks everyone for the discussion around the recent ESXi 7.0 U3k release.  We have heard you and are working on qualifying the ESXi 7.0 U3k patch for SimpliVity.  I will post back to this thread once the patch is qualify for SimpliVity. 

/Kipp

[ErwinH]

Hi Kipp,

Is there any news about a ETA for the ESXi 7.0 U3k release? A target date would be nice for maintenance planning.

Regards, Erwin

[Kipp_Glover]

Good day ErwinH!

The current plan is to have the ESXi 7.0 U3k qualified by March 21st, 2023.   Please remember that this plan could change. I will post an update once qualified or I hear of a change in plans.  

Cheers!
/Kipp

[ErwinH]

Hi Kipp,

is the 21st still achievable?

Regards, Erwin

[Kipp_Glover]

Good day Erwin!

The plan was to release a new SVTSP (firmware bundle) along with support for ESXi 7.0 U3k on March 21st.  We have had a delay with the SVTSP release.  I am pushing to announce the support for ESXi 7.0 U3k on the 21st.   I will post an update in this thread tomorrow.  

Cheers!
/Kipp

[Kipp_Glover]

Good day Erwin!

Qualification of ESXi 7.0 U3k has been completed for Simplivity.   Tomorrow we plan on posting an updated Interoperability Guide and a Customer Notice announcing the qualification of U3k.

The ESXi patch bundle from VMware can be applied via ESXCLI. The ESXi patch bundle can be downloaded from the VMware patch portal at https://customerconnect.vmware.com/patch

ESXi 7.0 U3k Supported versions:

• 4.1.1U1
• 4.1.2
• 4.1.3

ESXi 7.0 U3k Compatible Versions but not explicitly verified

• 4.1.0
• 4.1.0U1

For further details, consult the updated HPE SimpliVity OmniStack Interoperability Guide.

Cheers!
/Kipp

[ErwinH]

Hi Kipp,

Great, thanks for the update!

Kind regards,

Erwin

[jessiet1]

Thank you Kipp.  What are the steps to apply this successfully using ESXCLI?

[Kipp_Glover]

Good day All!

Today we published a new Interoperability Guide and we now support the ESXi 7.0 U3k patch. http://www.hpe.com/info/simplivity-interop-doc 

A customer notice will be published shortly. 

Cheers!
/Kipp

[Kipp_Glover]

Good day All!

The customer notice has been published announcing support the ESXi 7.0 U3k: 

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00130990en_us

Cheers!
/Kipp

[jessiet1]

Thank you!  This just says "The patch can be obtained from the VMware website and it can be installed manually." but doesn't specify what those steps are.  I assume I just

Upload patch to datastore

SSH into host

Run: esxcli software vib update --depot=/<path_to_vib_ZIP>/VMware-ESXi-7.0U3k-21313628-depot.zip

reboot

Are there any more steps or is this correct?

[jessiet1]

I just used update manager and that worked fine.

[Kipp_Glover]

Below are the steps to follow to apply the ESXi patch.

1. Upload the patch to the VMFS datastore (Not the SimpliVity datastore) on the target ESXi host. (Using WinSCP/directly from VCenter UI)
2. Migrate or power off all the virtual machines running on the target ESXi Host
3. Shutdown the OVC using HPE-SimpliVity safe shutdown option
4. Put the ESXi host into maintenance mode 
5. Access the ESXi host through SSH and then browse to the VMFS datastore that contains the zip file: (cd /vmfs/volumes/<target datastore>)
6. Run this esxcli cmd to update the patch: "esxcli software vib update -d "/vmfs/volumes/<target datastore name or id >/<patch-name.zip>"

Note- If the command worked then it will print this message: (The update completed successfully, but the system needs to be rebooted for the changes to be effective....)

1. Reboot the ESXi Host
2. Exit the host from maintenance mode
3. Verify the new build number from VCenter UI (Host summary page).
4. Verify that the OVC is running fine and check the VMs Storage HA status

Cheers!
/Kipp

[Per_Johansson]

Hello Kipp.

Thanks for the info regarding  ESXi 7.0 U3k

Kind Regards 

Per

[Crimson1]

How do we obtain the custom HPE ESXi 7.0 u3k image?  The latest available HPE custom image is 7.0 u3i.

[Kipp_Glover]

Good day Crimson1,

At this time we do not have a SimpliVity ESXi 7.0 U3k custom bundle.  The latest SimpliVity Custom Bundle is U3i.  You can apply the VMware U3k patch using the esxcli command. 

Cheers!
/Kipp

[Crimson1]

Will this work for an HPE ProLiant DL360 Gen10 (Not Simplivity)?