MSA 2052 & Vsphere 6.7 - discrepancy free space

[kr_andy]

Hi Community,

I'm looking at the available storage capacity in Vsphere and comparing it with the actually used space on the MSA 2052 Pool.
The whole Pool of the MSA is used as a VMFS6 volume by vsphere.

All vdisks in all VMs are Thick Proivisioned eager zeroed.
I have 6.54TB in total and 5.99TB used. So I have 566GB free.

If I check the MSA I can see 6.54TB in total but 2.16TB is avaiable unallocated.

I do understand the the discepancy. 
Vsphere reports used space based on the used vdissk which are thick provisioned.
MSA reports way more unallocated space because it's a virtual Array and data has not been written yet.

My question now is what happens if I use up the remaining 0.5TB in vsphere and vsphere tells me I need more space. The MSA Pool propably will still have 2TB left of unallocated space.
How can I utilize the thin provisiond virtual array of MSA so I don't have to buy new disks?

Best practice for thin provisioned virtual storage array used in vsphere is to use Thick provisioned vdisk for the VM. Right? But how can I now utilze this inside vsphere?

Am I missing a feature on MSA or vpshere side?
Am I understanding something the wrong way?

I just don't think it makes sense to buy new drives if I haven't reached a certain threshold only because vsphere doesn't "see" the unallocated space on the MSA.

Thank you in advance.

Regards,
Andy

[SUBHAJIT KHANBARMAN_1]

@kr_andy 

You need to provide more details like what is the total size of the MSA Volume, how much is showing as used space out of it? Also good to know Pool level space details from where this MSA volume belongs to as well as space details for all VDG part of this Pool.

Next thing same MSA volume after presented to ESX host and formatted with VMFS filesystem, also after creation of all VMs. Now what is the used space and free space showing at the datastore level?

When you present a MSA volume to ESX host and use it as datastore means formatted this MSA volume with VMFS filesystem. Now if you have created one or multiple VMs that is the moment vmdk files will gets created. As you have mentioned all VMs Thick provision eager-zeroed which means for MSA all vmdk files are fully allocated space. MSA don't understand VMware architecture. MSA firmware will check VMFS filesystem level and if any space zeroed out then it will start reclaim that space. In your case that possibility not there until you delete the entire VM or the vmdk file.

So when you compare or check space from MSA then you need to check MSA volume used and free space with ESX datastore used space and free space only. 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[kr_andy]

If I log into MSA 2050 I see this Under Home:

Capacity
Logical:                                                                            Reserved:3175GiB Allocated: 4462GiB Unallocated: 2235Gib
Physical: Unused: 1490GiB Global Spares: 1676GiB     Virtual Disk Groups: 9873GiB

Storage A
Virtual: 1 Volumes, 0 Snapshots
Allocated: 4462GiB Unallocated: 2235Gib
Virtual Pool: 6698GiB

This is what I see for the Vsphere datastrore
Capacity: 6.54TB
Used: 5.99 TB
Free: 564.78GB

If I login to MSA and selcet Action-- > Change Pool settings
I can see the dialog and "Enable overcommitment of pool?" has a greyed out tick.

Are you saying that the MSA already reclaimed the zeroed out space but vsphere will only recalim and see that if I delete the corrosponding vdisk?

So how can I utilise overcommiting of the pool on the MSA inside vsphere? Would I need thin provisioning vdisks or can it still work with thick provisioned disks? 

[SUBHAJIT KHANBARMAN_1]

Physical Capacity of the System
-------------------------------------------
Total Physical Disk Capacity of the MSA = (9873 + 1490 + 1676) GiB = 13039 GiB

VDG Space = 9873 GiB
Global Spare = 1676 GiB
Unused space = 1490 GiB >>>>> This is the space getting wasted here and no use of it

-------------------------------------------------------------------------------------------------------------

Logical Capacity of the System
----------------------------------------

Total Logical capacity = (3175 + 4462 + 2235) GiB = 9872 GiB

Reserved Space = 3175 GiB
Allocated = 4462 Gib
Unallocated = 2235 GiB

-------------------------------------------------------------------------------------

Pool A Capacity
-------------------

Total Usable space = 6698 GiB

Allocated = 4462 Gib
Unallocated = 2235 GiB

-----------------------------------------------------------------------------------

It seems overcommit is enabled at the Pool level so we are dealing with Thin volume already at the MSA. In order to confirm this you can use command "show pools" and look for column named "OverCommit" and it should show "Enabled".

You haven't mentioned Volume size in MSA which is most important information missing here.

Can you please provide the same ? It's good to have "show volumes" command output to get clear idea

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[kr_andy]

That physical unused space is the hot spare.
See output for pools, volume, disk groups and disks:

# show pools
Name Serial Number Blocksize Total Size Avail Snap Size OverCommit Disk Groups Volumes Low Thresh
Mid Thresh High Thresh Sec Fmt Health Reason Action
---------------------------------------------------------------------------------------------------------------------------
A xxx 512 6698.0GiB 2254.3GiB 0iB Enabled 2 1 50.00 %
75.00 % 97.01 % 512e OK
---------------------------------------------------------------------------------------------------------------------------

# show volumes
Pool Name Total Size Alloc Size Type Health Reason Action
------------------------------------------------------------------
A MSA2050-SAS 6698.0GiB 4443.7GiB base OK
------------------------------------------------------------------

# show disk-groups
Name Blocksize Size Free Pool Tier % of Pool Own Pref RAID Disks Spr Chk Status Jobs
Job% Serial Number Spin Down SD Delay Sec Fmt Health Reason
Action
----------------------------------------------------------------------------------------------------------------------
SAS-dgA01 512 6698.0GiB 2254.3GiB A Standard 100 A A RAID5 5 0 512k FTOL VRSC
36% xxx Disabled 0 512e OK

rcA1 512 1486.9GiB 0iB A Read Cache 0 A A NRAID 1 0 N/A UP
xxx Disabled 0 512e OK

----------------------------------------------------------------------------------------------------------------------

# show disks
Location Serial Number Vendor Rev Description Usage Jobs Speed (kr/min) Size Sec Fmt
Disk Group Pool Tier Health
--------------------------------------------------------------------------------------------------------------------
1.1 xxx HP HPD6 SSD SAS VIRTUAL POOL 0 1490.4GiB 512e
rcA1 A Read Cache OK
1.2 xxxHP HPD6 SSD SAS AVAIL 0 1490.4GiB 512e
N/A OK
1.9 xxxHP HPD2 SAS VIRTUAL POOL VRSC 10 1676.7GiB 512e
SAS-dgA01 A Standard OK
1.10 xxxHP HPD2 SAS VIRTUAL POOL VRSC 10 1676.7GiB 512e
SAS-dgA01 A Standard OK
1.11 xxHP HPD2 SAS VIRTUAL POOL VRSC 10 1676.7GiB 512e
SAS-dgA01 A Standard OK
1.12 xxxHP HPD2 SAS VIRTUAL POOL VRSC 10 1676.7GiB 512e
SAS-dgA01 A Standard OK
1.13 xxxHP HPD2 SAS VIRTUAL POOL VRSC 10 1676.7GiB 512e
SAS-dgA01 A Standard OK
1.23 xxxHP HPD2 SAS GLOBAL SP 10 1676.7GiB 512e
N/A OK
--------------------------------------------------------------------------------------------------------------------

[SUBHAJIT KHANBARMAN_1]

@kr_andy 

Thank you for sharing the outputs but remove serial numbers from the outputs as it's not good to share serial numbers in public forum.

Please note you have one SSD drive size of 1490 GiB at location 1.2 which is in AVAIL state which means not used by anyone. This is the space reflecting as Physical Unused space.

You have Global spare configured at location 1.23 which is size of 1676.7 GiB.

One important point that I want to mention that Host Operating systems usually show volume size in base 2. Disk drives usually show size in base 10 so in this case if you want to compare size then you need to check MSA side what is set. In general MSA side it should be set as base 10 which you need to modify as base 2. In SMU look for Managing Users section and then look for "Base Preference",

• Base Preference. Select the base for entry and display of storage-space sizes:
 Base 2. Sizes are shown as powers of 2, using 1024 as a divisor for each magnitude.
 Base 10. Sizes are shown as powers of 10, using 1000 as a divisor for each magnitude. This is a default.

Now coming to Vsphere datastrore,
Capacity: 6.54TB
Used: 5.99 TB
Free: 564.78GB

Right now MSA volume shows 2254.3 GiB free but ESX datastore shows 567.78 GB free which is clearly not matching. So two things you need to do. First set MSA user base preference from base 10 to base 2. Next you need to login to ESX CLI and run unmap command for that datastore in order to do space reclaim. After that we should check how much showing as used space in MSA volume and same way how much space shows as used in datastore used size. Both should be same.

Please note eventhough ESX 6.7 supports automatic space reclaim but it will not work with MSA because MSA works with 4 MB pages only. In VMware ESXi 6.5 and above there is a limitation of if any array unmap granularity is greater than 1MB then automatic unmap not supported. For VMFS6, reclamation granularity equals the block size. When you specify the block size as 1 MB, the granularity is also 1 MB. Storage sectors of the size smaller than 1 MB are not reclaimed. (You can also check with your VMware admin or support)

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[kr_andy]

Thanks for the reply.

I'm already aware of Base 2vs10 and already switched to base 2.

I ran the folowing command on one of the ESXi hosts cli:
esxcli storage vmfs unmap -u 5c5c5xxx-7291e540-xxx-5480xxx23b90

It took a while and finished. No output.
But nothing changed in the display of storage usage.

What do you mean by this:
"Please note eventhough ESX 6.7 supports automatic space reclaim but it will not work with MSA because MSA works with 4 MB pages only. In VMware ESXi 6.5 and above there is a limitation of if any array unmap granularity is greater than 1MB then automatic unmap not supported. For VMFS6, reclamation granularity equals the block size. When you specify the block size as 1 MB, the granularity is also 1 MB. Storage sectors of the size smaller than 1 MB are not reclaimed. (You can also check with your VMware admin or support)"

Is this sometihng I need to check on VmWare side?
Is it even possible to change something like this on the running volume?
Or would this only be the explanation of my discrapancy?
Do you maybe know any VMWare KB that would explain this?

Thank oyu.

[SUBHAJIT KHANBARMAN_1]

@kr_andy 

You can refer the below HPE Support Community forum on the automatic space reclaim topic and it was that time ESX 6.5 but same logic still applies for ESX 6.7 as well,

https://community.hpe.com/t5/msa-storage/msa2042-unmap-on-esx6-5/td-p/6970665#.XzKzDigzaUk

Now coming to below command that you have executed,

"
I ran the folowing command on one of the ESXi hosts cli:
esxcli storage vmfs unmap -u 5c5c5xxx-7291e540-xxx-5480xxx23b90
"

Is "UUID: 5c5c5xxx-7291e540-xxx-5480xxx23b90" represents MSA volume named "MSA2050-SAS" what we are dealing with?

If MSA base and ESX base both are base 2 then it's really an issue but I can't comment further until we check MSA volume "MSA2050-SAS" in ESX host at the datastore level in live system cause we need to understand what all vmdk files accumulating 5.99TB size in ESX host which is actually consuming 4443.7GiB in MSA volume side.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[kr_andy]

Yes "UUID: 5c5c5xxx-7291e540-xxx-5480xxx23b90" represents MSA volume named "MSA2050-SAS".
(x's are from me)
It's what I see under the datstore in vsphere:

• Typ:VMFS 6
  
• URL:ds:///vmfs/volumes/5c5c5357-7291e540-3b67-548028523b90/

The Setup is pretty straight forward.
All diskgroups are for the Pool A, which uses up the whole volume and the VMFS6 volume is created on the whole MSA volume.

[SUBHAJIT KHANBARMAN_1]

@kr_andy 

Unable to access "URL:ds:///vmfs/volumes/5c5c5357-7291e540-3b67-548028523b90/"

Better if you share screenshot as image file.

If ESX end MSA2050-SAS volume shows as 5.99TB as used compared to 4443.7GiB which is at the MSA end then better to open a support case with HPE as well as with VMware.

As per my understanding I don't see any issue at the MSA side because GUI and Command line both shows volume used space as 4443.7GiB at MSA end with base 2

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************