topic Re: MSA 2060 virtual pool running out of available space doing nothing in MSA Storage

MSA 2060 virtual pool running out of available space doing nothing

Agesa — Wed, 30 Nov 2022 10:41:06 GMT

Hi everyone!

I have 2 virtual pool in my system. In one of then, dedicated to video recordings, I am running out of space and I dont know why.

My virtualization system is VMWare 7 and my datastorage assigned to that pool keeps in the same size that I created with. No changes there.

But when I log in to manage the msa, i see that everyday I loose some GB of available storage.

This is my actual situation:

Pool allocation

Base volume allocated: 28.3 TB / 98.6%

Snapshot Allocated: 0%

Total commited: 400.5GB / 1.4%

Tiering Capacity

Archive 0%

Standar

Allocated 28.3 TB / 98.6%

Available 400.5 GB/ 1.4%

Disks: 16

Performance: 0%

Yesterday´s situation:

This is my actual situation:

Pool allocation

Base volume allocated: 28.3 TB / 98.4%

Snapshot Allocated: 0%

Total commited: 419.5GB / 1.4%

Tiering Capacity

Archive 0%

Standar

Allocated 28.3 TB / 98.4%

Available 419.5 GB/ 1.4%

Disks: 16

Performance: 0%

Tiering capacity is also running out like "Pool allocation Total commited".

Is this a normal behaviour due to tiering or something like that?

Overcommit is enabled in that pool also. Just to let you know.

Thank you in advance, I am quite newbie with this things.

Re: MSA 2060 virtual pool running out of available space doing nothing

SUBHAJIT KHANBARMAN_1 — Fri, 25 Nov 2022 17:13:47 GMT

@Agesa

You need to share details of Pool size and Volume size where video recordings gets stored . Also share size details of volume which is presented in VMware where video recordings gets stored.

You have shared only volume allocation details but we need to understand provisioned size both at MSA and VMware as well.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA 2060 virtual pool running out of available space doing nothing

Agesa — Sun, 27 Nov 2022 09:10:42 GMT

Hi!

sorry, I will try to share that information here:

My VMWare says that my video storage´s capacity is 26,14 TB, and I have used 26TB. This is like that since I created it a couple a months ago. This available space (140GB) will never change (and has never changed) as it is the space we defined for the recording virtual machine hard disks, and that is unchangeable.

About my pool, the total space defined there is 28,7TB. I think that this is the info you asked for.

If something else is needed, please, let me know

Thank you!

Re: MSA 2060 virtual pool running out of available space doing nothing

ArunKKR — Mon, 28 Nov 2022 09:22:37 GMT

Hi,

Could you launch an SSH session to controller management IP and execute the below commands and share the output?

show pools

show volumes

show volumes details

Please share the name of the Pool and volume as well.

Re: MSA 2060 virtual pool running out of available space doing nothing

Agesa — Mon, 28 Nov 2022 09:33:22 GMT

Hi!

Here it is:

HPE MSA Storage MSA 2060 FC
System Name: Uninitialized Name
System Location: Uninitialized Location
Version: IN110P001
# show pools
Name Serial Number Blocksize Total Size Avail Snap Size
OverCommit Disk Groups Volumes Low Thresh Mid Thresh High Thresh
Sec Fmt Health
------------------------------------------------------------------------------
A 00c0ff65c98b000058ffcb6201000000 512 9,5TB 6017,0GB 0B
Habilitado 1 1 50.00 % 75.00 % 97.75 %
512e Correcto
B 00c0ff65ca8e000098ffcb6201000000 512 28,7TB 370,5GB 0B
Habilitado 2 1 50.00 % 75.00 % 99.25 %
512e Correcto
------------------------------------------------------------------------------
Success: El comando se completó correctamente. (2022-11-28 11:25:00)
# show volumes
Pool Name Total Size Alloc Size Type Health Reason Action
----------------------------------------------------------------------------
A Sistemas_Operativos 9,5TB 3557,4GB base Correcto
B Video 28,7TB 28,3TB base Correcto
----------------------------------------------------------------------------
Success: El comando se completó correctamente. (2022-11-28 11:26:39)
# show volumes details
Pool Name Total Size Alloc Size
Serial Number WR Policy Cache Opt
Read Ahead Type Tier Affinity Snap Retention Priority Desc
WWN Large Virtual Extents Role
Metadata In Use Health Reason Action
-------------------------------------------------------------------------------
A Sistemas_Operativos 9,5TB 3557,4GB
00c0ff65c98b0000ab00cc6201000000 escritura no simultánea estándar
Adaptable base Sin afinidad medio
600C0FF00065C98BAB00CC6201000000 Deshabilitado
11,2MB Correcto
B Video 28,7TB 28,3TB
00c0ff65ca8e00002f01cc6201000000 escritura no simultánea estándar
Adaptable base Sin afinidad medio
600C0FF00065CA8E2F01CC6201000000 Deshabilitado
54,3MB Correcto
-------------------------------------------------------------------------------
Success: El comando se completó correctamente. (2022-11-28 11:27:16)
#

Re: MSA 2060 virtual pool running out of available space doing nothing

SUBHAJIT KHANBARMAN_1 — Mon, 28 Nov 2022 10:06:19 GMT

@Agesa

It looks like your Pool B is working as main storage for your Video recording. Your Pool B total capacity 28.7 TB and you have created 28.7TB volume as well. You have created thin volume at MSA end and out of 28.7TB, approx 28.3TB got used.

Now question is what type of volume you have created at the VMware end. It looks like you have created Thick provision volume (A Lazy Zeroed disk or Eager Zeroed disk). This is the reason most of the thin provisioned space of volume in MSA already got allocated. Do you think you have stored 26TB of video in this volume in real ?

Your Volume size shows 28.7TB as provisioned size in MSA and at the VMware side you should see same size only but you are seeing 26.14TB. If size different then your unit base may not be same. Operating systems usually show volume size in base 2. Disk drives usually show size in base 10. So change the base at MSA end and then compare size again.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA 2060 virtual pool running out of available space doing nothing

Agesa — Mon, 28 Nov 2022 10:27:38 GMT

Hi again,

first of all, thank you for your help.

Yes, my video recordigns are using those 26TB, I have a process that records 24x7 some cameras, every night some files are erased to create space for the new ones of the day coming.

But my problem is that despite the size of the datastorage in VMWAre is not increasing, as I wrote before, it is always 26,14 TB, my free space (Total commited) is running to 0 slowly.

Should I be worried about that? I dont mind if it reaches 0 and nothing happens, I will not increase my video datastorage in VMware.

I dont want either to overcommit anything.

Re: MSA 2060 virtual pool running out of available space doing nothing

SUBHAJIT KHANBARMAN_1 — Mon, 28 Nov 2022 11:06:51 GMT

@Agesa

I can see that even if video files getting deleted and new videos getting recreated in same space of VMFS but some block still getting consumed at MSA which should be really concern for you.

I am not sure which version of VMware you are dealing with but it seems auto space reclaim feature or auto unmap feature is enabled. Interesting thing to notice here at MSA end 4-MB pages gets reclaimed so you need to check what is the VMFS block size set. If it's set as 1-MB then auto reclaim will not work. You can check the VMware link to get more details,

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.doc/GUID-BC1A172C-E649-4812-B8B2-A9E45AC97051.html

You can manually try space reclaim or initiate space reclaim at VMFS level after video files gets deleted at VMFS level and check if that helps you. Please consult with your VMware admin before you try this as this may create performance issue.

Coming to MSA end, you check "show disk-group-statistics" outout and look for columns "Pages Deallocated per Min" and "Pages Reclaimed". This will give you idea that space getting reclaimed back at MSA end.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA 2060 virtual pool running out of available space doing nothing

ArunKKR — Mon, 28 Nov 2022 11:46:56 GMT

Hi,

28.7TB volume has been created from MSA storage and presented to host server.
While creating a data store you have the option to either use entire MSA volume capacity or any size lesser than MSA volume size.
I believe that entire MSA volume size would have been used to create data store in your configuration.
28.7 TB converts to 26.1 TiB
MSA by default represents size in TB and operating system in TiB.
You would need to monitor the free space of the command show pools.
Current status is 370.5 GB free space in Pool B.
You need to ensure that free space does not fall under 200GB.
As this is a thin provisioned Pool if any data is deleted from OS end the free space does not reflect in MSA immediately.
Space reclamation needs to happen in the backend for the actual free space to show up on MSA and this takes time.
Leaving just about 370 GB space and continuing daily back-ups might lead to Pool high threshold of 99.25% exceeding.
Exceeding Pool high threshold results in controller cache moving to write through mode and degraded performance.
I would suggest adding a new disk group to Pool B by adding drives to avoid the risk of Pool getting full/exceeding high threshold.

Re: MSA 2060 virtual pool running out of available space doing nothing

Agesa — Mon, 28 Nov 2022 12:57:47 GMT

Hi!

We have vmware 7, and auto space reclaim is enabled. How do I check the block size in my vmfs?

I still dont get why my vmdk files stay the same size, my datastorage stays the same size, but I run out of that space in the msa.

Thanks a lot!!!

Re: MSA 2060 virtual pool running out of available space doing nothing

SUBHAJIT KHANBARMAN_1 — Mon, 28 Nov 2022 15:08:26 GMT

@Agesa

Please find the steps,

Log in to the ESX console.
Run the command: # vmkfstools -P <path to datastore> The block size is in bold in this line: Capacity 429228294144 (409344 file blocks * 1048576), 8896118784 (8484 blocks) available. Where 1048576 is equivalent to a 1 MB block size on a VMFS datastore.

I would suggest to first match the provisioned size (base 2 or base 10) both at MSA and VMware end then only you will be able to understand actual space usage.

Also initiate unmap manually at VMware end and check if you are getting space reclaimed or not.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA 2060 virtual pool running out of available space doing nothing

Agesa — Tue, 29 Nov 2022 08:45:45 GMT

Thanks, I will try to do it.