HPE 3PAR StoreServ Storage

3PAR RAW Space 100% Full, compactcpg possible?

 
SOLVED
Go to solution
jwe
Advisor

3PAR RAW Space 100% Full, compactcpg possible?

Hi,

customer reports that their 3PAR 8200s RAW Space is 100% full. There is still some free space in the used CPG (about 1TB). Some more CPG space will be free soon, when several decomissioned VMs have been removed and unmapped.

Is it possible to run compactcpg in this situation to free up some raw space capacity?

regards

5 REPLIES 5

Re: 3PAR RAW Space 100% Full, compactcpg possible?

@jwe 

Yes you can check all CPGs and try compactcpg if any space reclaim possible or not. In order to check check that you can try below command which actually run in dryrun mode which will show you if any space reclaim opportunity for that CPG,

compactcpg -dr <CPG_name>

If above command shows any opportunity then you can go for below one CPG at a time,

compactcpg -f <CPG_name>

If you are planning to decomissioned VMs (I am assuming ESX environment) then yes 1st thing to do initiate space reclaim process from ESX filesystem level. As we don't know what is the VMWare version so can't comment which process to follow. However, if it's higher than ESX 5.0 but less than 6.5 then unmap command will work. If ESX 5.0 then vmkfstools will work. If ESX6.5 and above then space reclaim should happen automatically.

At 3PAR end you need to check 3PAR VV Usr reserved space and Usr used space. User used space should match with VmWare VMFS datastore used size after unmap of VMFS level space reclaim finish. 3PAR VV usr reserved space will not come down immediately. Explanation given below why. So you need to keep checking all CPGs and if any space reclaim opportunity found you can compact the CPG. Otherwise you need to look for deleting snapshots and VVs directly from block level to free up space.

Some guidelines how space reclaim works in 3PAR for tpvv,

Thin Persistence reclamation

Thin Persistence reclamation occurs at several levels. Initially all freed 16 KiB pages are returned to the TPVV. This means that on a file system that supports automatic reclaim the spaced freed by an UNMAP after a file deletion is immediately available for reuse by a file creation or file extension operation on the same file system.

To make space available for reuse by other volumes, there is a reclaim thread that returns freed 128 MiB regions allocated to a TPVV back to the CPG. This thread scans volumes every five minutes for SD space that potentially can be reclaimed. If a TPVV has free 128 MiB regions or there is enough free space to warrant a defragmentation of the volume then space will be reclaimed back to the CPG. Defragmentation occurs if there is more than 1 GB of available space to reclaim and results in free 128 MiB regions. Up to 16 volumes at a time can be queued for reclaim processing.

How quickly the space is reclaimed depends on a number of factors. If there is a large amount of freed space on a volume, then this may not be processed within a single reclaim period and once the reclaim process runs on a TPVV, the reclaim process will not run again on that TPVV again for at least 90 minutes. Therefore, large space reclaims can take several hours to complete.

In addition, the reclamation on a TPVV can be deferred for various reasons.

For example, if the SD space of a TPVV is grown, then reclaims on the volume will be deferred for 60 minutes. Also, if reclaim is de-fragmenting a TPVV and the defragmentation does not complete during the reclaim interval further reclaim will be deferred for 8 hours.

Thin Persistence reclamation may not reclaim all the free space on a volume. There is a 4 GB per node threshold below which the TPVV will not be inspected for available 128 MiB regions that can be reclaimed back to the CPG. The free space will still be available for reuse by the TPVV. Those new to Thin Provisioning often like to verify Thin Persistence reclamation by creating a test scenario of filling a file system then deleting the files and running a space reclamation tool. It is important to understand that the space will not be returned to the CPG immediately. The showvv –s command will show how much space has been allocated to the TPVV and the difference between the space in use and the reserved space shows the amount of space reclaimed for use within the TPVV. The amount of reserved space will decrease over time as the space is reclaimed back to the CPG in the background by the reclaim thread.

 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************


I work for HPE
Accept or Kudo
jwe
Advisor

Re: 3PAR RAW Space 100% Full, compactcpg possible?

@Subhajit,
Fine, I wasn't aware that compactcpg has a "-dr"-option. I will give it a try. VMware is version 6.5, but there is a mixture of VMFS5 and VMFS6 volumes. We are using "esxcli storage vmfs unmap" for VMFS5 datstores and we are monitoring with "showvv -s" the reserved space of the virtual volumes (tpvv).

Solution

Re: 3PAR RAW Space 100% Full, compactcpg possible?

Yes you are correct.

esxcli storage filesystem list

esxcli storage vmfs unmap –l <datastore_name>

or

esxcli storage vmfs unmap –u <UUID_datastore>

From 3PAR end yes you need to keep checking showvv -s and compactcpg -dr option but reserved space will come down slowly as explined in last post.


If you have no further query and you are satisfied with answers then kindly mark the forum as resolved.


Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

 


I work for HPE
Accept or Kudo
jwe
Advisor

Re: 3PAR RAW Space 100% Full, compactcpg possible?

One more question:

Should we try the option "-trimonly" with compactcpg in the first step?

Re: 3PAR RAW Space 100% Full, compactcpg possible?

As per my understanding if you execute compactcpg -dr <CPG_name> then initially it will show you exclusive unused LD only. After that it will show total LD space to be reclaim. You can crossverify by running below commands,

compactcpg -dr -trimonly <CPG_name>

compactcpg -dr <CPG_name>

You can see below example,

XXX cli% compactcpg -dr -trimonly NL_r6
Compacting CPG NL_r6
Initial trim would reclaim 0 MB SA space, 28672 MB SD space, and 0 MB USR space.
4096 MB from LD tp-3-sd-0.2
8192 MB from LD tp-3-sd-0.1
12288 MB from LD tp-3-sd-0.4
4096 MB from LD tp-3-sd-0.0

XXX cli% compactcpg -dr NL_r6
Compacting CPG NL_r6
Initial trim would reclaim 0 MB SA space, 28672 MB SD space, and 0 MB USR space.
4096 MB from LD tp-3-sd-0.2
8192 MB from LD tp-3-sd-0.1
12288 MB from LD tp-3-sd-0.4
4096 MB from LD tp-3-sd-0.0
Would move 0 MB SA space, 58880 MB SD space, and 0 MB USR space.
Final trim would reclaim 0 MB SA space, 262144 MB SD space, and 0 MB USR space.
24576 MB from LD tp-3-sd-0.2
57344 MB from LD tp-3-sd-0.5
32768 MB from LD tp-3-sd-0.1
49152 MB from LD tp-3-sd-0.4
40960 MB from LD tp-3-sd-0.0
57344 MB from LD tp-3-sd-0.3

 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************


I work for HPE
Accept or Kudo