Server 2016 bluescreen vmswitch.sys

[FelR]

I am facing a serious issue on two Hyper-V-hosts with Windows Server 2016 Datacenter.

Without external influence they crash and all guest VMs too. The STOP code is UNEXPECTED_KERNEL_MODE_TRAP 0x0000007f and it is caused by vmswitch.sys A simple reboot does not fix this, it always crashes 8 to 10 times in a row until it works properly again.

This problem occurs on two of our servers with completely different hardware. I tried several versions of the nic driver, but it did not resolve the issue. VMQ is already disabled on the nics and in the VM settings.

I updated all drivers und bios to the latest versions. In one server I changed the Ethernet adapter card, but the issue persisted.

Here you can download a recent minidump file: https://dl.dropboxusercontent.com/u/76615769/110416-20546-01.dmp

I analyzed a recent memory.dmp with WinDbg and the failure was caused by vmswitch!VmsPktParseIPv4Packet.

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind
that the kernel isn't allowed to have/catch (bound trap) or that
is always instant death (double fault).  The first number in the
bugcheck params is the number of the trap (8 = double fault, etc)
Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
        use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
        use .trap on that value
Else
        .trap on the appropriate frame will show where the trap was taken
        (on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 0000000000000008, EXCEPTION_DOUBLE_FAULT
Arg2: fffff80288d24e70
Arg3: fffff80288d0efc0
Arg4: fffff801d4a0fa0a

Debugging Details:
------------------


DUMP_CLASS: 1

DUMP_QUALIFIER: 400

BUILD_VERSION_STRING:  10.0.14393.351 (rs1_release_inmarket.161014-1755)

SYSTEM_MANUFACTURER:  HP

SYSTEM_PRODUCT_NAME:  ProLiant MicroServer Gen8

SYSTEM_SKU:  F9A40A          

BIOS_VENDOR:  HP

BIOS_VERSION:  J06

BIOS_DATE:  11/09/2013

DUMP_TYPE:  2

BUGCHECK_P1: 8

BUGCHECK_P2: fffff80288d24e70

BUGCHECK_P3: fffff80288d0efc0

BUGCHECK_P4: fffff801d4a0fa0a

BUGCHECK_STR:  0x7f_8

TRAP_FRAME:  fffff80288d24e70 -- (.trap 0xfffff80288d24e70)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffff62658c80bfb8 rbx=0000000000000000 rcx=ffff918a74f14c30
rdx=0000000000000065 rsi=0000000000000000 rdi=0000000000000000
rip=fffff801d4a0fa0a rsp=fffff80288d0efc0 rbp=0000000000000000
 r8=0000000000000366  r9=fffff80288d12cb0 r10=0000000000000000
r11=fffff801d4a57343 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
vmswitch!VmsPktParseIPv4Packet+0x1a:
fffff801`d4a0fa0a 4c89442438      mov     qword ptr [rsp+38h],r8 ss:0018:fffff802`88d0eff8=????????????????
Resetting default scope

CPU_COUNT: 4

CPU_MHZ: 8f7

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3a

CPU_STEPPING: 9

CPU_MICROCODE: 6,3a,9,0 (F,M,S,R)  SIG: 1B'00000000 (cache) 1B'00000000 (init)

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT_SERVER

PROCESS_NAME:  System

CURRENT_IRQL:  2

ANALYSIS_SESSION_HOST:  FELIX-WIN10

ANALYSIS_SESSION_TIME:  01-07-2017 14:10:13.0467

ANALYSIS_VERSION: 10.0.14321.1024 amd64fre

STACK_OVERFLOW: Stack Limit: fffff80288d0f000. Use (kF) and (!stackusage) to investigate stack usage.

STACKUSAGE_FUNCTION: The function at address 0xFFFFF801D4A734DC was blamed for the stack overflow. It is using 15264 bytes of stack total in 106 instances (likely recursion).

FOLLOWUP_IP: 
vmswitch!VmsPktParseIPv4Packet+63aec
fffff801`d4a734dc 90              nop

STACK_COMMAND:  .trap 0xfffff80288d24e70 ; kb

THREAD_SHA1_HASH_MOD_FUNC:  93424311f65cbfa2e614c0eb88b95d5b29167b57

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  dce58a582219dea1d7e594f3df9698be645a23a9

THREAD_SHA1_HASH_MOD:  cc4158ba2185e63d52742ed134310e999c22d8d5

FAULT_INSTR_CODE:  c685e990

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  vmswitch!VmsPktParseIPv4Packet+63aec

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: vmswitch

IMAGE_NAME:  vmswitch.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  57dacdf2

IMAGE_VERSION:  10.0.14393.206

BUCKET_ID_FUNC_OFFSET:  63aec

FAILURE_BUCKET_ID:  0x7f_8_STACK_USAGE_RECURSION_vmswitch!VmsPktParseIPv4Packet

BUCKET_ID:  0x7f_8_STACK_USAGE_RECURSION_vmswitch!VmsPktParseIPv4Packet

PRIMARY_PROBLEM_CLASS:  0x7f_8_STACK_USAGE_RECURSION_vmswitch!VmsPktParseIPv4Packet

TARGET_TIME:  2016-11-04T10:56:59.000Z

OSBUILD:  14393

OSSERVICEPACK:  351

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  400

PRODUCT_TYPE:  3

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 Server TerminalServer DataCenter SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2016-10-15 05:38:38

BUILDDATESTAMP_STR:  161014-1755

BUILDLAB_STR:  rs1_release_inmarket

BUILDOSVER_STR:  10.0.14393.351

ANALYSIS_SESSION_ELAPSED_TIME: 10d0

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x7f_8_stack_usage_recursion_vmswitch!vmspktparseipv4packet

FAILURE_ID_HASH:  {54616c76-71d3-7277-08ba-c2d2fa4a114c}

Here is the call stack:

nt!KeBugCheckEx
nt!KiBugCheckDispatch+0x69
nt!KiDoubleFaultAbort+0xb3
vmswitch!VmsPktParseIPv4Packet+0x1a
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParseIPv4Packet+0x63aec
vmswitch!VmsPktParsePacket+0x156
vmswitch!VmsNblGenerateRssHash+0xd3
vmswitch!VmsVmNicPvtPacketForward+0x234
vmswitch!VmsRouterDeliverNetBufferLists+0x81a
vmswitch!VmsExtPtReceiveNetBufferLists+0x193
NDIS!ndisMIndicateNetBufferListsToOpen+0x11e
NDIS!ndisMTopReceiveNetBufferLists+0x265fc
NDIS!ndisCallReceiveHandler+0x47
NDIS!NdisMIndicateReceiveNetBufferLists+0x735
vmswitch!VmsExtMpIndicatePackets+0x7b4
vmswitch!VmsExtMpSendNetBufferLists+0x5a8
NDIS!ndisMSendNBLToMiniportInternal+0xee
NDIS!NdisSendNetBufferLists+0x36c
vmswitch!VmsExtPtRouteNetBufferLists+0x3fe
vmswitch!VmsPtNicReceiveNetBufferLists+0x8a0
NDIS!ndisMIndicateNetBufferListsToOpen+0x11e
NDIS!NdisMIndicateReceiveNetBufferLists+0x26ca3
b57nd60a+0x48804

I already discussed this problem in the TechNet forum, but the tips there did not help. Maybe someone here can help me

Thanks for your efforts, FelR

[WilianC]

Hi,

Does anyone know how to fix this problem? I have exactly the same problem but I can`t find the solution.

I Updated firmware and driver for Broadcom/Qlogic Network adapter.

Any Idea?

[FelR]

Hi,

I ended up opening a call with MS. A temporary workaround was to disable TCP offloading using these commands

netsh int tcp set global rss=disabled
netsh int tcp set global autotuninglevel=disabled
netsh int tcp set global chimney=disabled

Finally the problem was fixed by a new version of vmswitch.sys in the KB4013429 update.