554FLB causes Hyper-V to freeze guests

[jdixon]

I've noticed an issue with some of my virtual machines running on a particular host after that host was fully patched on Windows and HPE drivers/firmware from the 8.1 HPE service pack.

What would happen is VM's would get stuck in a stopping state for some reason. The host would all of a sudden try to move all the VM's from it but majority of them would get completely stuck which results in my having to hard reboot the server.

At first I thought this may have been a Windows update issue with a bad patch, but after looking closer I think this is an issue with the 554FLB adapter we are using:

2/12/2018:

The network interface "Emulex OneConnect 554FLB, Ethernet NDIS Miniport Driver" has begun resetting. There will be a momentary disruption in network connectivity while the hardware resets.
Reason: The network driver did not respond to an OID request in a timely fashion.
This network interface has reset 1 time(s) since it was last initialized.

2/1/2018:

The network interface "Emulex OneConnect 554FLB, Ethernet NDIS Miniport Driver" has begun resetting. There will be a momentary disruption in network connectivity while the hardware resets.
Reason: The network driver did not respond to an OID request in a timely fashion.
This network interface has reset 4 time(s) since it was last initialized.

According to the event viewer, Hyper-V was running fine with no issues until this warning occurred. When this warning occurred it followed with an error for each VM on the host:

A component on the server did not respond in a timely fashion. This caused the cluster resource 'SCVMM ********************' (resource type 'Virtual Machine', DLL 'vmclusres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.

The firmware for the 554FLB is 11.1.183.23 and the driver on Server 2016 Datacenter is 11.1.196.4

Note: This is the only node in the cluster that is up to date with HPE drivers/firmware and Windows patches so we could test it first. This is also the only node having this issue.

[f_silva]

Hi

The messages indicate that tou have any issue related with drivers.
This nic work fine when the FW are correct to the Drivers version

The FW used in the NIC is old, it is very important that you use the

I saw in HPE that for 554FLB don't have drivers link available so is possible that nic isn't upported to Windows 2016, Also searching in Windows Catalog not found the 554FLB for Windows 2016 just for 2008 R2.

Windows Catalog: https://www.windowsservercatalog.com/results.aspx?text=Emulex+BE3&bCatID=1282&avc=10&ava=0&OR=5

I reccommend you use the last version availble of the Driver and Firmware.

[jdixon]

I noticed that too but I also find their new website to be very complicated. The BL460c Gen8 is completely supported with 2016 and I thought the 554FLB was. 

The driver sys 2016-08-27 Version 11.1.196.4. The downloads for Emulex (just says generic Emulex) under BL460c Gen8 is: https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=5177951&swItemId=MTX_4bd8a53ecce84591928cdba06b&swEnvOid=4184#tab1 which show 2016

[jdixon · Accepted solution]

I ended up discovering that this was not a driver issue with the NIC even though I was getting that warning. It turns out when I have hyper-v replication enabled, it causes this situation when live migrating. Disabling hyper-v replication and the issue is no longer. I opened a case with Microsoft but as usual it is taking FOREVER.

[JimPCI]

Did you ever find a solution or did Microsoft ever get back to you? Disabling Hyper-V replication is not an option.

We just experienced the same issue as you for the first time this past weekend - every VM was in a failed state on a host with no apparent reason. A hard reset of the host was the only way to resolve the issue.

I've seen Live Migration issues when Hyper-V hosts have a third party anti virus installed, but this appears to be a different issue.

[jdixon]

We stopped using it but we think it was: https://support.microsoft.com/en-us/help/4077525/windows-10-update-kb4077525

Specifically:

Addresses issue that causes Hyper-V VMs that are replicated using Hyper-V Replica or Azure Site Recovery to stop responding at 92% if a Windows Server 2016 Failover Cluster is set up with NIC Teaming enabled.  The issue also occurs while stopping the VM, during Live Migration, while stopping the VMMS service, or during Host node shutdown. The user must then use a hard restart on the host machine to recover.

After patching it does seem to have resolved the issue.

[JimPCI]

Thanks for the update. We're patched up to July so it must be a different issue.