PSOD VMWare ESXi 6.7 and HP StoreVirtual 12.6 on every reboot or shutdown

[erwinbre]

Hi all,

Our stable situation before: 6 ESXi v6.5 hosts running on HPE DL380 gen10 servers. All hosts are using our HP StoreVirtual v12.6 storage with StoreVirtual Multipathing Extension Module (MEM) for VMWare v6.5 installed.

New situation: Upgraded VSphere to v6.7 ( build 11726888) and upgraded 1 ESXi host to v6.7 (11675023). Also upgraded the StoreVirtual Multipathing Extension Module (MEM) for VMWare v6.7 on this host.

Problem: Every reboot or shutdown of this server ends with a PSOD (see attachement).

If we uninstall the StoreVirtual Multipathing Extension Module (MEM) and revert to the Vmware-Round Robin Path Selection Policy, reboot en shutdown works fine.

The problem must be with the StoreVirtual Multipathing Extension Module (MEM). We also tried the MEM v6.5 but that gave us the same PSOD.

I hope anyone can help us.

Thanks,

Erwin

[regi_p]

Hi i have a similler issue with my G10 after upgrading (No VSA) but temp fix from VMW https://kb.vmware.com/s/article/58874

Just letting you know this may help 

All the best 

[erwinbre]

Hi, thanks for answering.

The VMW article specifically references  'lb_netqueue_bal' .

Did you see this reference in your PSOD?

Thanks,

Erwin

[regi_p]

Hi see below reply from VMW & PSOD is attached - Hope you fix your issue 

- This is VMware ESXi 6.7.0 build-11675023
- Server Model and BIOS version
  HPE ProLiant DL380 Gen10 | BIOS: U30 | Date: 10/02/2018
 - CPU type and how many
  Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
- CPU microcode revision
  Current Revision:0x02000050
  
PSOD Traces : 
------------
RSSPlugCleanupRSSEngine@(lb_netqueue_bal)
RSSPlugInitRSSEngine@(lb_netqueue_bal)
RSSPlug_PreBalanceWork@(lb_netqueue_bal)
Lb_PreBalanceWork@(lb_netqueue_bal)
UplinkNetqueueBal_BalanceCB
UplinkAsyncProcessCallsHelperCB
HelperQueueFunc
CpuSched_StartWorld

Analysis : 
=======
Cause : 
-------
This issue occurs when the load-based netqueue balancer module, misses or fails to cleanup the RSS engine private data, as a result dedicated heap gets full and causes the crash during subsequent load-balancing.

Resolution : 
-----------

This is a known issue affecting ESXi 6.7.x. This issue will be resolved in the ESXi version 6.7 U2 which is due to release in early 2019.  

Hence, To workaround this issue, run this command  "esxcli network nic queue loadbalancer set --rsslb=false -n vmnicX" for all the vmnics on the Host. 

This change would not require the Host to be rebooted. 

Below is the KB article that talks about this issue. 

https://kb.vmware.com/s/article/58874 

This should take care of the issue.

[regi_p]

Also i have 4 ESX nodes all are same Nodes 1-3 runs latest VSA (Note: I don't use any MEM etc...) - No PSOD at all after VMW 6.7 update

the last node (No VSA) but had one PSOD after latewst VMW 6.7 update 

Since then i have done the VMW temp fix all are good 

[erwinbre]

Unfortunately the fix doesn't work for my problem, tested a few minutes ago.

I thought so because in our particular PSOD there is no reference for 'lb_netqueue_bal' .

This must be something with the MEM multipathing.

[Mukesh2]

Hi,

Apologies for the delayed response.

For detailed informaiton on PSOD, please contact VMware.

Having said that we believe the issue reported could be because of the version you running, as version VSA 12.6 on ESXI 6.7 is not advised.

Though its compatible we have seen ESXI 6.7 having a conflict with MEM version 6.5 running with 12.6.

We recommend you upgrade the VSA to 12.7 for better performance as it includes the latest patches including fixes for few issue reported earlier and has a different verison of MEM for use.

Action Plan :

1. Schedule a downtime for this activity.
2. Uninstall the current version of MEM you are running.
3. Reboot the hosts.
4. Download the MEM version for ESXI 6.7 using the shared below :

https://h20575.www2.hpe.com/ecommerce/efulfillment/getSWReceipt.do?lc=EN_US

5. Install this version on the hosts.
6. Reboot the Hosts.

Check if it works fine or still issues are seen. If the issues are still seen upgrade VSA version to 12.7.

Regards,
Mukesh