VMWare 7.0.3 C or E VVOL support bundle with 3par 8200 causes VVOL loss of communication

[Erik Wattnem]

Fyi,

  We had an issue after we went to Esxi 7.0.3 c from 6.7 for our HPE Synergy 480 hosts that had VVOL setup to our 3Par 8200. that I just want to make people aware of.  This was not a HPE issue but a bug with VMWare VVOL and ,in this case, a HPE 3par 8200.  We found if you run a VMWare support bundle with the "storage" default section checked on a Esxi 7.0.3c or Esxi 7.0.3e that it would cause the host to lose access to the VVOL volumes we had with the 3PAR 8200..   Some indicators was that if you had VM's on VVOLs they would show up with 0 used space in vcenter client and you couldn't vmotion the guest that was affected.  If you ssh'ed to the host and tried to a ls on the vvol you would say "no such device or address".  The VMFS volumes were ok.  The only way we found to fix it once that happened was to reboot the host.  We worked with VMWare and went all the way to Engineering.  They found this which I have not found documented anywhere and why I am listing it here to help others that may come across it.

Root Cause :

- An optimization (SCSIBlockUnsupportedOpcodesAndPages) to VMware’s Pluggable Storage Architecture (PSA) was introduced in ESXi 7.0 U3 which is designed to prevent issuing commands to Storage Devices after receiving a SCSI Command failure with an Invalid Operation Code.
- For the impacted vVol Datastores, we can see that Protocol Endpoint (PE) Devices previously failed SCSI Commands with error code “0x5 0x20” which translates to ILLEGAL REQUEST / INVALID COMMAND OPERATION CODE –
- Due to the newer SCSIBlockUnsupportedOpcodesAndPages optimization, these devices are flagged with an Unsupported status which is then cached persistently which means that all future SCSI command attempts will fail, thus resulting in the vVol Datastore Read/Write failures we observed.

Resolution: Issue is fixed in ESXi version 7.0 Update 3f 20036589. Vmware recommend upgrading one of the host to 7.0 Update 3f 20036589 and re-run the log collection test.

There is a "workaround" that can do to avoid going to F or later but I suggest that you talk to VMWare to make sure it is ok for your environment.

[support_s]

System recommended content:

1. VMware vVols with HPE Alletra 9000 Storage | Number of vVols per VM

2. HPE ESXi Offline Bundle

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[sbhat09]

Hello @Erik Wattnem,

Thank you very much for sharing this. Certainly useful for the larger audience if anyone facing the same issue.

Regards,
Srinivas Bhat

[Erik Wattnem · Accepted solution]

Thanks Srinivas!

Here is the link to the VMWare KB article fresh off the presses.

Virtual machine residing on vVol datastores go unresponsive during log collection. (89919) (vmware.com)