3PAR StoreServ Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

 
Highlighted
YYCSysAdmin
Frequent Advisor

Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

Hello everyone, I come to you with 2 new questions today.

I have been running UNMAP against our front end datastores in order to continue our consolidation and reclamation of space while slowly migrating to VMFS6 front end datastores.

It appears that the remote copy group volumes on our other 3PAR are not updating or shrinking now that the master volume has dropped in size, in some cases by up to 50% or more. I presumed that the shrinking would have replicated over to the .r volume?

Do i need to stop the RCG , remove the .r copy and let the RCG recreate it in order to see that freed up space from the UNMAP running?

My second issue. Also with running UNMAP, i noticed that the Reserved Space on a Virtual Volume has not dropped either. As per below, there is 1.4TB of space being reserved that is not been released since running UNAMAP. I know i could do a convert to Full (they are currently thin) and then convert it back to Thin, but that seems very time consuming and would involve me shuffling around more things and doing Tune to move some volumes to new tiers with more space to allow me to convert to full.

Example Volume:

Capacity Utilization 48% Total used
Total used 1,945.84 GiB
Virtual size 4,096.00 GiB
Total reserved 3,430.13 GiB
Host written 0.00 GiB

6 REPLIES 6
Highlighted

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

UNMAP command in ESX side helps to reduce used space of VV in 3PAR and not Reserved space of VV.

You can check this detail,

***********************************Thin Persistence reclamation-------------------------------

Thin Persistence reclamation occurs at several levels. Initially all freed 16 KiB pages are returned to the TPVV. This means that on a file system that supports automatic reclaim the spaced freed by an UNMAP after a file deletion is immediately available for reuse by a file creation or file extension operation on the same file system.

To make space available for reuse by other volumes, there is a reclaim thread that returns freed 128 MiB regions allocated to a TPVV back to the CPG. This thread scans volumes every five minutes for SD space that potentially can be reclaimed. If a TPVV has free 128 MiB regions or there is enough free space to warrant a defragmentation of the volume then space will be reclaimed back to the CPG. Defragmentation occurs if there is more than 1 GB of available space to reclaim and results in free 128 MiB regions. Up to 16 volumes at a time can be queued for reclaim processing.

How quickly the space is reclaimed depends on a number of factors. If there is a large amount of freed space on a volume, then this may not be processed within a single reclaim period and once the reclaim process runs on a TPVV, the reclaim process will not run again on that TPVV again for at least 90 minutes. Therefore, large space reclaims can take several hours to complete.

In addition, the reclamation on a TPVV can be deferred for various reasons.

For example, if the SD space of a TPVV is grown, then reclaims on the volume will be deferred for 60 minutes. Also, if reclaim is de-fragmenting a TPVV and the defragmentation does not complete during the reclaim interval further reclaim will be deferred for 8 hours.

Thin Persistence reclamation may not reclaim all the free space on a volume. There is a 4 GB per node threshold below which the TPVV will not be inspected for available 128 MiB regions that can be reclaimed back to the CPG. The free space will still be available for reuse by the TPVV. Those new to Thin Provisioning often like to verify Thin Persistence reclamation by creating a test scenario of filling a file system then deleting the files and running a space reclamation tool. It is important to understand that the space will not be returned to the CPG immediately. The showvv –s command will show how much space has been allocated to the TPVV and the difference between the space in use and the reserved space shows the amount of space reclaimed for use within the TPVV. The amount of reserved space will decrease over time as the space is reclaimed back to the CPG in the background by the reclaim thread.

***************************************************************************************************************************************

You can check from "HP 3PAR Thin Technologies whitepaper”,

https://h20195.www2.hpe.com/v2/Getdocument.aspx?docname=4AA3-8987ENW

“Blocks of 16 KB of contiguous space filled with zeros are freed and returned for reuse by the VV;

blocks of 128 MB of contiguous space are freed and returned to the CPG for use by other volumes.

Ie., For the “Reserved User Size” to reduce, the contiguous area of 128MB would be needed.

With HP 3PAR OS 3.1.1 and higher, the system automatically runs a defragmentation process on the array on a per-VV basis that concatenates areas of 16 KB of zeros in the volume to a contiguous area of 128 MB.

This enables greater space reclamation within the CPG and the array.”

If the volume is heavily fragmented then for the reserved space to be returned to the system, the defrag process required. This is a background process and can be deferred based on the fragmentation level. Due to the requirement of contiguous area of 128MB, this is a slow process and could take several hours or days.

So if you see that the difference between Usr Rsvd and Usr Used is more – so defrag process will reduce the Rsvd size automatically.

 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**********************************************************************


I work for HPE
Accept or Kudo
Highlighted
YYCSysAdmin
Frequent Advisor

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

Thank you for that great info. I presume there is no command for me to force free up that Reserved space?

Here is another volume that i had used UNMAP on last week Monday. So it has been over a week but the Reserved space has not change. I have run compactcpg at least 10 or more times as well as a compactld just incase also, but on change.

So far it seems like none of the Virtual Volume I have run UNMAP against, have had the Reserved space amount changed at all.

Total used 1,725.91 GiB

Virtual size 4,096.00 GiB

 Total reserved 3,251.00 GiB

Host written0.00 GiB


Just for info we are running 2 x 7400 series with SSD/FC/NL

  1. HPE 3PAR OS version
  2. Current 
    3.2.2.709 (MU6)+P99,P107,P119,P131,P135,P137,P138
Highlighted

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

You can use the below command to check per day if reserved space or Used space comes down or not as below,

srvvspace -daily <VV_name>

There are some HPE internal commands which may help. You can also log case with HPE Support and we will help you accordingly

 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

**********************************************************************


I work for HPE
Accept or Kudo
Highlighted
YYCSysAdmin
Frequent Advisor

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

Hello,

This volume i ran UNMAP on it 4 days ago

2019-08-04

Since then the reserved size has not changed from 3515648 in size.

I do have a ticket in with HPE but they noted there is nothing that can be done except wait and update to 149 as patch 144 has some optimizations. But i do wonder if after 4 days nothing has changed on the 8 volumes I have run UNMAP on it seems like something is not running at all to get this space back.

 

srvspace-daily.PNG

 If the picture doesnt show right:

 

                                   --------RawRsvd(MB)-------- -------User(MB)-------- -------Snap(MB)-------- -----Admin(MB)------ ---------------Total(MB)--------------- -Capacity Efficiency-
                   Time       Secs    User  Snap Admin   Total    Used    Free    Rsvd  Used  Free  Rsvd Vcopy Used Free Rsvd Vcopy Vcopy    Used    Rsvd VirtualSize   Compaction    Dedup

2019-07-30 00:00:00 MDT 1564466400 4788818 24384  8064 4821266 3467680   28512 3496192     0 16256 16256     0 2291  397 2688     0     0 3469971 3515136     4194304          1.2        -
2019-07-31 00:00:00 MDT 1564552800 4790525 24384  8064 4822973 3467148   29044 3496192     0 16256 16256     0 2291  397 2688     0     0 3469439 3515136     4194304          1.2        -
2019-08-01 00:00:00 MDT 1564639200 4787645 24384  8064 4820093 3468554   27638 3496192     0 16256 16256     0 2291  397 2688     0     0 3470845 3515136     4194304          1.2        -
2019-08-02 00:00:00 MDT 1564725600 4792509 24384  8064 4824957 3496192       0 3496192 16256     0 16256     0 2688    0 2688     0     0 3515136 3515136     4194304          1.2        -

2019-08-03 00:00:00 MDT 1564812000 4775740 24384  8832 4808956 3023873  472319 3496192     0 16256 16256     0 2615  329 2944     0     0 3026488 3515392     4194304          1.4        -

2019-08-04 00:00:00 MDT 1564898400 4766972 24384  9600 4800956 1990148 1506044 3496192     0 16256 16256     0 2294  906 3200     0     0 1992442 3515648     4194304          2.1        -

2019-08-05 00:00:00 MDT 1564984800 4766972 24384  9600 4800956 1990490 1505702 3496192     0 16256 16256     0 2294  906 3200     0     0 1992784 3515648     4194304          2.1        -

2019-08-06 00:00:00 MDT 1565071200 4834450 29376 10176 4874002 1992612 1503580 3496192     0 16256 16256     0 2294  906 3200     0     0 1994906 3515648     4194304          2.1        -

2019-08-07 00:00:00 MDT 1565157600 4768722 24085  9600 4802407 1988444 1507748 3496192     0 16256 16256     0 2294  906 3200     0     0 1990738 3515648     4194304          2.1        -

 

 

Highlighted

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

Thanks for sharing the Iinstalled nform OS version and patches. I can see that Patch 144 is not installed where space reclaim fix is addressed.

You can check the below release notes to refer the same,

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00072641en_us&docLocale=en_US

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00072844en_us&docLocale=en_US

I also suggest to apply Patch 149 to get the fix.

As informed you earlier, UNMAP command helps to reduce the Used size which is successfully done I can see but reserved space not reduced for which you need to have patch 149 as this patch obsoletes patch 144

 

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*********************************************************************


I work for HPE
Accept or Kudo
Highlighted
YYCSysAdmin
Frequent Advisor

Re: Virtual Volumes - reserved space not shrinking after UNMAP or RCG .r disks?

Hello,

Hopefully as early as tomorrow morning we may have patch 149 installed.