Hyper-V 2022 - Unable to move VMs between DL360 G9 and DL360 G10.

[ERoberge]

Hi,

We are currently using Microsoft 2022 Hyper-V with 2 Servers (HV-04 = DL360G9 and HV-03 = DL360G10) and 1 SAN MSA2062FC and we are unable to move VMs between these servers.
Even if we enable the option "Migrate to a physical computer with a different processor version", we are unable to move the VM between the two hosts.

Error Message: The Virtual machine cannot be moved to the destination computer. The hardware on the destination computer is not compatible with the hardware requirements of this virtual machine.
VM "APPS01" uses CPU-specific features that are not supported on the physical machine
“HV-04”. To enable migration of this virtual machine to physical machines with different processors, modify the
virtual machine settings to limit the CPU features used by the virtual machine.

CPU and Bios version :

DL360G10 : Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz 2.19 GHz (1CPU)
SMBIOSBIOSVersion : U32
Manufacturer : HPE
Name : U32
SerialNumber : **confidential info erased**
Version : HPE - 1

DL360 G9 : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz 2.40 GHz (2 CPUs)
SMBIOSBIOSVersion : P89
Manufacturer : HP
Name : P89
SerialNumber : ***Confidential info erased***
Version : HP - 1

We can shut down the VM from HV-03 and boot manually on the second Hyper-V Host HV-04 without any problem.
Do you know how I can enable this option (moving VM between servers)?

I really appreciate any help you can provide.

Eric R.

[support_s]

System recommended content:

1. HPESBHF04243 rev.3 - Certain HPE Servers using certain Intel Processor BIOS, INTEL-SA-00527, 2021.2 IPU – Intel BIOS Advisory, Multiple Local Vulnerabilities

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[shiva_jr]

Hi  ERoberge,

As per my knowledge, if both server processor are from same vendor(Intel or AMD) migration has to work. Here DL380 servers have Intel processors. 

Try the following steps-

1) Run the Powershell command in source HyperV server to veiw the Compatibility for Migration mode is enabled.

get-VMProcessor

2) Run these Powershell commands to configure the Compatibility for Migration mode to enable.

get-vm -name <vm_name> | stop-vm

Set-VMProcessor <vm_name>  -CompatibilityForMigrationEnabled $true

start-vm -name <vm_name>

Regards,

SHIVA_JR

Please mark as 'Accepted solution' if my post worked.

[ERoberge]

@shiva_jr 

Thanks SHIVA_JR,

as you can see below, all the VMs already have the "CPU Compatibility mode" activated.

get-VMProcessor * -computername hv-03

VMName | Count | CompatibilityForMigrationEnabled | CompatibilityForOlderOperatingSystemsEnabled

VM1 | 4 | True | False
VM2 | 2 | True | False
VM3 | 2 | True | False
VM4 | 2 | True | False
VM5 | 2 | True | False
VM6 | 2 | True | False
VM7 | 2 | True | False
VM8 | 1 | True | False
VM9 | 4 | True | False

Unfortunately, we cannot move our VMs between the two Hyper-V Servers.

I'm still trying to find a solution...

[Vinky_99]

@ERoberge 

Good day! 

You may give a try to the below instructions: 

* Shut down the VM on HV-03: Make sure the VM is powered off gracefully before proceeding.

* Open the Hyper-V Manager on HV-03: Launch the Hyper-V Manager and navigate to the settings of the VM you want to move.

* Edit the VM's settings: In the settings window, go to the Processor section. Here, you need to limit the CPU features used by the virtual machine to ensure compatibility with the destination host.

* Review the CPU-specific features enabled for the VM and disable any that are not supported on HV-04. This could include features such as virtualization extensions or specific instruction sets.>> Once you have made the necessary changes, save the VM settings.

* try to migrate the VM to HV-04 again using Live Migration or Quick Migration. This time, the migration should proceed successfully as the VM's CPU features have been adjusted to match the capabilities of the destination host.

By limiting the CPU features used by the virtual machine, you ensure compatibility with the destination host and allow for successful VM migration between the DL360 G9 and DL360 G10 servers.

I hope this gives some insights. Let me know how it goes....

[ERoberge]

Hi @Vinky_99 

Thanks for your reply,  this is the result of our tests.

The configuration:
HV-04 = DL360 G9 (Old server)
HV-03 = DL360 G10 (New server)

VM CPU Settings :
Processor compatibility settings :
Migrate to a physical computer with a different processor version: ENABLE.

Processor NUMA Settings :
Max number of processors: 12
Max amount of memory: 63476 MB
Max NUMA nodes allowed on a socket: 1
Hardware threads per core: 2
(I remove the default value of "0" and use the "2 threads per core" to disable the inherit the host settings for hardware threads per core).

What I tried :
If I boot the VM from HV-04 (Old server), I can do a live motion between Hosts without any issue (Move to HV-03 and move it back HV-04). == Working perfectly.
If I boot the VM from HV-03 (New server), I cannot do a live motion between hosts. == Not working.

The workaround:
I need to boot the VMs from HV-04 (Old server) and do a live motion to HV-03 (New server).
This way, if I need to do maintenance on HV-03 (New server), I can use live motion to migrate the VMs back to HV-04 (Old Server). 

Next steps :
I'll continue to search for the solution to do Live Motion regardless of the host server used to start the VM.
ANY hints are welcome!

Best regards,

ERoberge.

[Vinky_99]

Thank you for providing the additional information and the results of your tests. Based on the details you've shared, it seems that the live migration issue is specific to VMs that are initially booted on HV-03 (DL360 G10, the new server) and then attempting to live migrate them to HV-04 (DL360 G9, the old server). However, the live migration works fine when the VM is initially booted on HV-04 and then migrated to HV-03.

Here are a few suggestions to troubleshoot the live migration issue and potentially find a solution:

1. Ensure that both servers (DL360 G9 and DL360 G10) are running the latest BIOS and firmware versions.

2.Review the BIOS settings on both servers and ensure they are configured similarly, especially in areas related to virtualization, CPU features, and power management. Look for any significant differences between the BIOS settings of the two servers that could impact live migration.

3. Double-check the CPU compatibility between the servers. Although the "Migrate to a physical computer with a different processor version" option is enabled, it's worth confirming that the CPU architectures and features of the processors on both servers are indeed compatible. You can refer to the processor specifications (Refer documentation) or contact HPE support for further clarification.

4. Ensure that the Hyper-V integration components are up to date on both the VMs and the host servers. Outdated or incompatible integration components can sometimes cause migration issues. Update the integration components on the VMs and the Hyper-V host servers to the latest version.

* Check the event logs on both servers (HV-03 and HV-04) for any relevant error messages or warnings related to live migration. Look for any specific error codes or detailed messages that might provide insight into the root cause of the issue.

* If the issue persists after trying the above steps, it's advisable to reach out to HPE support for further assistance.

It's worth noting that live migration between different hardware generations can sometimes be challenging due to differences in CPU architectures and features. However, with the proper settings and compatibility, it should be possible to achieve successful live migration between your DL360 G9 and DL360 G10 servers.

[ERoberge]

Thank you @Vinky_99 

Unfortunately, we only have a valid support contract for our new server and our SAN. we don't have a valid support contract for our old server (DL360 G9) and we are not able to access the latest SSP for this server.

But I will try to double-check every parameters/settings and I will get back with the result at the beginning of July.

Thanks again for your time!

ERoberge.

[t-ict1]

Hi all,

We are experiencing the exact same problem. We even have a somewhat older gen10 (let's call this a gen10v1) which also cannot receive VM's which are started on the newer gen10v2 servers.

When the VM is started on the gen10v1, it is possible to live migrate them to the gen9. So the latest gen10 servers have a certain capability on the CPU which prevents the live migration.

If I find something, i'll let you know. If you have a resolution, please let me know  

[ERoberge]

 @t-ict1  Thanks for the information. If I find something, I will update my post.

Since we don't have an active support contract with our old server, we don't have access to the latest updates (firmware, bios, etc).

But you confirm that the problem is not bound to the updates...

I really hope to find something!

Good luck.

[t-ict1]

Yes confirmed. all the Hyper-V hosts have the latest Service Pack on them.

[Joseph00]

Any body got a solution to this problem?

[ERoberge]

Unfortunatly not at this time for me.

[Joseph00]

Thank you for the response, So are you still  following the pattern,like power on the VM from DL 380 Gen 9 and then the live migration works

[dalbjerg]

Does any have any news update this.
I have this error aswell. And the checkmark is set, and the machine have been rebooted.

[Joseph00]

Hello All,

Live migration did not work between Gen 10: U30 and Gen 9 :P89

for us downgrading the System ROM fw helped, but not recommended as the latest is always recommended but with that LM did not work

ProLiant DL380 Gen10 System ROM U30 - till version 2.62 it worked ,above that did not

This is the link that helped us

https://learn.microsoft.com/en-us/answers/questions/1182701/hyperv-live-migration-between-2-differents-intel-p?page=1#answers

[ERoberge]

Thank you for sharing this information, we hope HP get back with a solution...

https://learn.microsoft.com/en-us/answers/questions/1182701/hyperv-live-migration-between-2-differents-intel-p?page=1&orderby=newest&comment=answer-1324734#newest-answer-comment:~:text=HPE%20got%20back,any)%20remains%20unknown.

[rhEngiV]

Same problem with DL380 G10 and DL380 G11.

With the newest BIOS (v3.20 for G10 [U30]/ 2.22 for G11 [U54]) Live-Migration is not working in both directions.

With BIOS v2.62 or lower on the G10 Server Live-Migration is working, if the VM is started from the G10 Server. If the VM is started on the G11 Server Live-Migration does not work.

Solution for now: Starting and Rebooting VMs only on the G10 Server and then Live-Migrating them to the G11 Server. If done like that the VMs can be Live-Migrated back to the G10 Server.

Offical answer from HPE Support (After many hours and mails...):

"...below is the statement of the senior engineers:

We are working closely with Microsoft and Intel on resolving this issue. We believe that the problem is not unique to HPE. It is a microcode change from Intel that caused the rejection of the live migrations by the processor features compatibility check between the hosts of the VM in WS2022 and WS2019.

It was noted that the priority set is high."

I will give an update if there is anything new from HPE Support...

[pmfna44]

Hi all

on the same page here, Gen10 servers with Hyper-v2019 Bios 3.20 and trying to move to another hyperv cluster 2022 with Gen11

Is there any workarround that not involves to bios downgrade on the Ge10 ?

thanks

[mikkar]

Plese keep us posted. I've a similiar case with HPE and they don't have a solution and blame Intel and MS: 

Our general recommendation will be to use the latest BIOS to avoid known issues - but since this prevents live migrations then this may not be possible for the customer - but running with an earlier BIOS would mean reduced security, and risk of hitting something already fixed.  
Choosing which would be a customer decision based on their own needs.

[rhEngiV]

New Update: Intel has fixed the Microcode in the CPU. Microsoft is working on an update for Windows Server 2022, to fix the bug. The bug is registered under the following number:

"Microsoft Bug 52601547: VM live migration failure due to CLX proc bios/uCode update"

Haven't gotten an answer from MS till now, when the update will be ready...

[mikkar]

Do you know if MS are working on a fix for Win Server 2019 as well?

[rhEngiV]

I don't know, if they are working on a fix for WS2019...

[mikkar]

Today, Microsoft has informed us that per HPE request, a new section "Failed to live migrate a VM between nodes with different microcode (uCode) revisions" was added to the troubleshooting article:
https://learn.microsoft.com/en-us/troubleshoot/windows-server/virtualization/troubleshoot-live-migration-issues

Please feel free to share this updated Microsoft article with our customers to inform them that this issue is not related to HPE BIOS.

There is a behavior change in Windows Server 2025 where VM Live Migration can be done from New to Old or vice vesa without any issue. This is because the dynamic processor compatibility mode is updated in WS2025 to take advantage of the new processor capabilities. However, Microsoft does not want to backport this change to WS2022

[T-ICT]

Oke and what about:

New Update: Intel has fixed the Microcode in the CPU. Microsoft is working on an update for Windows Server 2022, to fix the bug. The bug is registered under the following number:

"Microsoft Bug 52601547: VM live migration failure due to CLX proc bios/uCode update"

Second:

....to inform them that this issue is not related to HPE BIOS

How is this not related to HPE BIOS? Appareantly thie migration from new to old works with an older version of HPE bios? See earlier comments.