Microserver Gen 10 Plus Server 2019 Network Code 12

[dandickson]

I have 6 identically configured HP Microserver Gen10 Plus units with the following configuration:

- Xeon E-2224 processor
- 32GB memory
- 2 x 2TB SSD in RAID1 running on the S100i
- Intel X710-DA2 Adapter
- Windows Server 2019 HyperV Core

The system is patched to the latest 2020.03.0 SPP and has cp044092 and cp044239 applied post SPP
Additionally, the latest firmware was applied to the X710 and the latest Proset drivers.

All systems show some network adapters with Code 12 error, more commonly it's the pair of X710 ports however I have atleast 1 system showing the X710 ports working but one of the i350 ports in Code 12.

Reinstalling the Proset drivers will resolve the issue, however once the system is restarted the Code 12 issue returns.

Any suggestions? I'm at a loss as to where to even start with this issue.

Thanks

[Pra_Sah]

Hello dandickson,

Thank you for sharing your query.

Please share the error screenshot, for the network card i350 so that we can check assist you further on this query.

Regards,

[dandickson]

I cannot easily give you a screenshot showing the code 12 as I temporarily pushed the entire stack to Xen.

I'm specifically not using SR-IOV under Xen, when I enable SR-IOV even on a single port the system throws errors about not having enough free resources (exact error is: not enough MMIO resources for SR-IOV) (ref: https://support.citrix.com/article/CTX250121)

In about a week i'll have the test unit from the stack back on the bench and it wil be available to run whatever diagnostics you wish, unfortunatly the project hit a critical milestone and I need to deploy what I have so-far.

Thanks

[Pra_Sah]

Hello dandickson,

Thank you for the update.

Since the timeline is one week. Please confirm once you have the server available on your test bench so that we can assist you further. You can send me a private message using following steps
>Click on the user's  Profile name     
>This takes you to the members Profile Page, then click on Send Private Message button
>A new page opens for you to compose your message

Regards,

[dandickson]

@Pra_Sah 

I built out the test system using the desktop experience on 2019 to make troubleshooting easier.

- Post install LOM 3 and 4 were disabled
- Enabling LOM 3 and 4 worked and survived a reboot
- Adding vSwitches works correctly and show working however adapters will code 12 post reboot

For sanity, here's the powershell run, Embedded LOM 1 Port 1 (P1-1) is the port being used for management currently:

#Logically name physical ports:
Rename-NetAdapter -Name "Embedded LOM 1 Port 1" -NewName "P1-1"
Rename-NetAdapter -Name "Embedded LOM 1 Port 2" -NewName "P1-2"
Rename-NetAdapter -Name "Embedded LOM 1 Port 3" -NewName "P1-3"
Rename-NetAdapter -Name "Embedded LOM 1 Port 4" -NewName "P1-4"
Rename-NetAdapter -Name "PCIe Slot 1 Port 1" -NewName "P10-1"
Rename-NetAdapter -Name "PCIe Slot 1 Port 2" -NewName "P10-2"

#Create vSwitches
New-VMSwitch -Name "V1-2" -NetAdapterName "P1-2" -AllowManagementOS $false -EnableIov $true
New-VMSwitch -Name "V1-3" -NetAdapterName "P1-3" -AllowManagementOS $false -EnableIov $true
New-VMSwitch -Name "V1-4" -NetAdapterName "P1-4" -AllowManagementOS $false -EnableIov $true
New-VMSwitch -Name "V10-1" -NetAdapterName "P10-1" -AllowManagementOS $false -EnableIov $true
New-VMSwitch -Name "V10-2" -NetAdapterName "P10-2" -AllowManagementOS $false -EnableIov $true

Screenshots of process:

Post install LOM 3 and 4 disabledEnabled LOM 3 and 4 without errorCreate vSwitches without errorPost reboot code 12 errors

[dandickson]

@Pra_Sah 

I ran some more troubleshooting from the scope of it being an issue with SR-IOV.

I removed all vSwitches using Get-VMSwitch | Remove-VMSwitch, this caused all nework adaptors to show as onlne however post reboot both X710 ports showed as code 12

I set the VFs on all adaptors to 4 (default on the i350 is 6 and the X710 is 128), this caused all nework adaptors to show as onlne however post reboot both X710 ports showed as code 12. It's worth noting that running Get-NetworkAdapterSriov shows MissingAcs under SriovSupport.

I disabled SR-IOV on the X710 ports, post reboot both ports showed as code 12.

Thanks

[Pra_Sah]

Hello dandickson,

Thank you for the detailed  explanations.

I checked further and Intel® Ethernet Converged Network Adapters X710 is not listed as HPE provided device for this server. So i would request to contact Card Vendor so that they can assist you with this query as it looks like a Driver or firmware issue with the card.

Please confirm if you have tested the setup with any of the following recommended cards as per server quickspecs:- 

HPE Ethernet 1Gb 4-port BASE-T I350-T4 Adapter P21106-B21
HPE Ethernet 10Gb 2-port BASE-T QL41132HLRJ Adapter P08437-B21
HPE Ethernet 10Gb 2-port SFP+ MCX4121A-XCHT Adapter P21930-B21

Regards,

[dandickson]

@Pra_Sah 

We were using the X710-DA2 cards for serveral reasons:

- We already had them
- They are one of the most compatible cards on the market
- The vendor matches the LOM implementation simplifying the required drivers for the deployment image

I'll skip the hilarity of using an Intel based LOM but not certifying any of the 10G+ Intel cards, it's worth internal discussion from HPs perspective given the lower end chassis implementation, but I don't feel we'll get to a solution from an external perspective. I'll rely on the confidence that HP actually tested these cards with SR-IOV and they will initialize correctly despite most of my research showing this is an issue in the way IOMMU is implemented in the system firmware.

I ordered a batch of P21930-B21 but HP has a 7-9 day lead from my VAR so it will be a couple of weeks before we're ready to test again, i'll update this thread when I do.

Thanks

[LTSS]

Hi dear,

How is going on ?

My Intel X550's SR-IOV doesn't work with Microserver Gen 10 Plus Server either.

[dandickson]

@Pra_Sah @LTSS 

Switched out for the genuine HP Mellanox part, same issue.

[Pra_Sah]

Hello dandickson,

Thank you for the update. This would require further investigation into the Driver and OS configuration. So, I would request to please log a support ticket with our Technical Team so that we can take this forward.

Regards,

[mysy]

I think HPE disable SR-IOV function from all basic module ML or DL server. Not the card issue. Bios have disable SR-IOV function.

I have same issue in my DL20 G9 and G10 Server, same X520 NIC card work in ML110 G9 server with SR-IOV, but not in DL20.

In DL20 G10 manual, it already point out not support SR-IOV.