MSA2042 Unmap on ESX6.5

[Qmonteyne]

Hi,
I hope anyone here is running a setup similar to mine, because I can't figure this out based on the documentation and I can't live-test this right now.

Setup:
*ESXi 6.5 (full patched soft & firmware) on HPE DL360p's
*HPE MSA2042 w/ one-to-latest firmware (008)
*Fibre Channel over Brocade Fabric switches, QLogic HBA's (HPE variant) in the hosts

*all thin provisioned VMFS volumes on MSA2042

*all datastores in VMFS6 (thin prov), and all datastores have the auto-unmap turned on (by default Low prio).

Issue:
*the deletion of VM's doesn't free the space on the Volume and thus on the Disk Groups. It shows free in the VMFS. To get the space back in the SAN, I need to empty & delete the volume.

*VAAI counters such as MBDEL in ESXTOP remain at 0 unless I manually run "esxcli storage vmfs unmap" on the datastore, which recovers some but not all the space. More importantly, I cannot run this anymore as it seems to have had such a performance impact a few datastores went down (APD?!?!) so no testing this before confirmation by VMware I can run it safely.
*!!! I should not need to run it

Note: If you are using VMFS6 in ESXi 6.5, this article is not applicable."

*the MSA2042's best practices sheet (https://www.hpe.com/h20195/v2/GetPDF.aspx/4AA4-6892ENW.pdf) clearly states VAAI and T10 unmap compatibility

*the vmware HCL lists the 2042 as capable of: "VAAI-Block Thin Provisioning, HW Assisted Locking, Full Copy, Block Zero" VMW HCL SAN

*according to this article by Cormac Hogan, the HCL should list a "footnote" to support Automatic Unmap

Question: should I be expecting auto-unmap on this array?

Vmware support pointed me to HPE. On ESXi side, everything is ok (VAAI support detected, enabled, etc.)

ps: support case id 17508974807

thank you
Quentin

[SUBHAJIT KHANBARMAN_1]

MSA Page size is 4MB and in VMware ESXi 6.5 there is a limitation of if any array unmap granularity is greater than 1MB then automatic unmap not supported. For VMFS6, reclamation granularity equals the block size. When you specify the block size as 1 MB, the granularity is also 1 MB. Storage sectors of the size smaller than 1 MB are not reclaimed. Please find the MSA best practice guide to understand Block size/Page size and also the VMWare link to understand how unmap and space reclaim works,

https://www.hpe.com/h20195/v2/getpdf.aspx/4aa4-7060enw.pdf (page 51)

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-BC1A172C-E649-4812-B8B2-A9E45AC97051.html

[SUBHAJIT KHANBARMAN_1]

in order to check space reclaim and unmap inside MSA then use command,

 show disk-group-statistics

[disk-group disk-group]

[type linear|virtual]

There are few parameters in the output which give more details on this,

 Pages Allocated per Min

Shown for a virtual disk group. The rate, in pages per minute, at which pages are allocated to volumes in the disk group because they need more space to store data.

 Pages Deallocated per Min

Shown for a virtual disk group. The rate, in pages per minute, at which pages are deallocated from volumes in the disk group because they no longer need the space to store data.

 Pages Reclaimed

Shown for a virtual disk group. The number of 4-MB pages that have been automatically reclaimed and deallocated because they are empty (they contain only zeroes for data).

 Pages Unmapped per Minute

Shown for a virtual disk group. The number of 4-MB pages that host systems have unmapped per minute, through use of the SCSI UNMAP command, to free storage space as a result of deleting files or formatting volumes on the host.

More details customer can refer Command Line reference guide (page no 287),

 http://h20564.www2.hpe.com/hpsc/doc/public/display?docLocale=en_US&docId=emr_na-c04957376

[Qmonteyne]

Hi  SUBHAJIT KHANBARMAN,

I believe you're the L2 engineer working on my case at HPE support :) ?

thanks for your feedback.  I've replied asking for confirmation of some stuff, and will be confirming with VMWare as well.

VMFS6 indeed uses a 1MB block size.

If I read this correctly, you’re confirming the MSA needs 4MB unmaps, which ESXi does not provide through the Auto Unmap functionality because 4MB is bigger than 1MB.

Therefore, Given the block size of VMFS6 is a fixed 1MB, and the MSA’s page size (and as such, the “optimal unmap granularity”) is a fixed 4MB, this can not work as auto unmap at this point in time?

[SUBHAJIT KHANBARMAN_1 · Accepted solution]

Your understanding is correct. There is no issue from MSA side and that's why manual unmap works but automatic unmap not working due to limitation at the Vmware side which is clearly mentioned in the VMware link.

If there is any issue at the MSA side then unmap or space reclaim will never work which is not our situation.

[Qmonteyne]

Thanks!

I agree, and this has given me something to feedback to VMware which i've done today. (remember, upon first support ticket they said "everything OK on our end, talk to your storage vendor" ....

While I can accept this being like it is technically, the interface and documentation should be much clearer about it. Hope this evolves in upcoming releases!

thank you,

Quentin