Reclaim space quickly (sdelete alternative)

[david_tan2]

This is related to the previous post outlining why pre-Windows2012 does not return space back to the array. I just can't seem to add a tag for "scriptingblitz"

https://connect.nimblestorage.com/message/1862#1862

There are two main problems with sdelete that I have observed:

1. It does not recognise mountpoints, so you need to add a letter to the mountpoint to run.

2. It is very, very slow.... taking sometimes days to reclaim a couple of TB.

Chris Duck came up with a great idea of using powershell to perform the same kind of task as sdelete, but has the advantage of working with mountpoints - http://blog.whatsupduck.net/2012/03/powershell-alternative-to-sdelete.html

I have pinch this idea and some of his code to create my own version that runs roughly 10x faster. It uses the same method to create a 1GB initial file (of zeros) which takes about 60 sec. From there I just use good old DOS copy to clone the file until there is no more space left of the drive (ie <1GB). The copy commands are surprisingly fast! and on the 10GB connected array it will reclaim at a rate of 200-300MB/sec, which is roughly 1TB/hr. There could be ways to squeeze more performance out of it using other copy utilities that allow multithreads like robocopy but this works well enough for my liking

Basic Usage is for example

powershell .\NimbleFastReclaim.ps1 E:\

powershell .\NimbleFastReclaim.ps1 E:\dir1\mountpointishere\

It will try to log to C:\temp dir but can specify different file. A third parameter can be passed to sleep between copies to throttle the process if desired.

[Daniel-san]

looks interesting but getting errors that it's not signed and i'm not up on powershell.  says execution of scripts disabled.  any idea how to make it run to test it without screwing around with signatures?  it does not say it's blocked so can't seem to even run it unsigned.

[aherbert23]

The default policy for windows is to disable scrips from running. You can set the execution policy to allow it. Run PowerShell as an administrator and type the following command.

Set-ExecutionPolicy bypass

[david_tan2]

Launch powershell prompt and check your settings

Get-ExecutionPolicy

I think the default is RemoteSigned. Change it to Unrestricted first

Set-ExecutionPolicy Unrestricted

Failing that use Bypass instead of Unrestricted. This is the technet reference for Powershell 3.0 but also applies for 2.0

http://technet.microsoft.com/en-us/library/hh849812%28v=wps.620%29.aspx

[Daniel-san]

nice - thank you.  glad one of us knows P/S!

[Daniel-san]

Thanks David.  So it worked well.  Looks like it took several hours on a 2TB volume so haven't been able to confirm a 10x speed increase but it does appear to work.

[a_user57]

Not to hijack this sled, but how are you reclaiming this space once you run the script? My understanding is that running sdelete (or similar) on a volume will fill the available space with zero's, in essence turning a thin provisioned volume into a thick provisioned volume. So if this is one vmdk of many on a vmfs volume, obviously the vmkfstools/unmap command will not apply to the vmdk (only vmfs), that has now expanded in size as a result of the script. So you need to vmotion this vmdk file to a different data store with a different block size, as thin provisioned then in order to reclaim that space and reduce the size of the actual vmdk file, correct?

[david_tan2]

Hi C W,

I should have mentioned that I am performing this task on an iSCSI presented volume rather than a VMDK. I have not tried it on a VMDK and I cannot actually test this myself, but the concept might be the same?? When data is written to the disk and then deleted nothing actually changes on disk except the reference to the file. The array therefore stills sees it as used data. Overwriting this with zeros (and of course deleting the file) allows the array to see this as free space again. Be interested to know if the vmfs in between storage and OS makes any difference here...maybe someone has tried it and can comment?

[david_tan2]

I have to admit my 10x was a finger in the air estimate based on approx IO rate

I used this on CS460x2 with 10GB network.

Are you on 10GB network? It should take roughly 2.5hrs for 2TB at a guess. Probably wouldn't matter too much on the Nimble Controller as all the 0's should be compressed very efficiently.

[marktheblue45]

Hi David,

              Testing this on a guest connected iSCSI Nimble thin provisioned volume optimised for SQL Logs. Copied 10GB of ISO files. Guest reports 10GB used and Nimble says the same ish.. Deleted the ISO files and ran your script. BINGO!  Within 10 minutes the array reported the space was down by 10GB. Genuinely impressed. Nimble could do worse than engage with you.

Kind Regards,

                       An extremely impressed Nimbler.

[Daniel-san]

My test was performed on an iSCSI guest-connected volume - not a VMDK.  We're using a CS220GX2 with 10G connections - seemed like it took >2.5 hours.

[marktheblue45]

Hi David, I created a new vmfs volume on nimble and added a Datastore through vcenter. Added Vm disk to a windows 2008R2 VM and put 2GB of files into it. Deleted them and ran your script. BINGO once again. Works with vSphere 5.5.

[a_user57]

What command can one use to run the script against remote servers? I would like to run this task against more than one server, once per month for example.

[david_tan2]

Probably a few ways to do it but I would use invoke-command

try something like this

invoke-command -filepath <path>\NimbleFastReclaim.ps1 -computername $ComputerName -ArgumentList "<pathtorun>"

You can also specify a list of computers in the parameter if they all have consistent paths you want to run on. Can also pass variables to the argumentlist if you want to run programatically - this might help http://stackoverflow.com/questions/4225748/how-do-i-pass-named-parameters-with-invoke-command

[a_user57]

Hi David,

Thanks for your assistance thus far.  When I try to perform the script, I get a failure as indicated below, any ideas?:

PS C:\Users\labadmin> Invoke-command -filepath C:\NimbleFastReclaim.ps1 -ComputerName test-fs1 -ArgumentList "e:" -Credential domain1.lab\Administrator

[2014-02-04 21:14.20] --> Starting Reclaim on e:\ ...

[2014-02-04 21:14.20] --> Cycle Sleep = 0 sec

[2014-02-04 21:14.20] --> File Size = 1024 MB

[2014-02-04 21:14.20]   -->Writing e:\NimbleFastReclaim0.tmp

[2014-02-04 21:14.20] ##> Reclaim Failed. Cleaning up...

The user account being used is a domain admin account, and is able to run the script locally on the server being specified in this example. Any help you or anyone else can provide would be great. I realize we are starting to deviate off-topic a bit, but I thought this might come in handy for others who might face a similar problem. My powershell experience is very limited.

[epedersen22]

Unfortunately I'm getting the same error on all my servers:

[2014-02-04 21:14.20] --> Starting Reclaim on F:\ ...

[2014-02-04 21:14.20] --> Cycle Sleep = 0 sec

[2014-02-04 21:14.20] --> File Size = 1024 MB

[2014-02-04 21:14.20]   -->Writing F:\NimbleFastReclaim0.tmp

[2014-02-04 21:14.20] ##> Reclaim Failed. Cleaning up...

Cheers,

Eric

[a_user57]

Glad to see this is not just me. I am sure we will be able to collectively come up with a solution. Unfortunately my knowledge of powershell is extremely limited, so I am of limited help, but it would be great to have an automated solution to handle all servers in a scripted and scheduled manner. I cant believe there are not others who would benefit from the same thing.

[david_tan2]

I'll try this out today and let you know. Will probably need some error handling so it spits out a useful error message. Btw - have you tried to run the invoke-command to a remote server with a simple command?

Eg invoke-command -computername <remotecomp> -scriptblock {get-wmiobject win32_computersystem}

[epedersen22]

Howdy,

Yes, I was able to run your example invoke-command, once I ran ‘WinRM quickconfig –q’ on the target server. However, still no joy with either local drives or mapped remote drives.

Eric

[david_tan2]

Hi Guys,

Turned out to be a memory error. Running powershell script remotely has finite memory limits depending on WinRM settings. This version should fix it by writing smaller chunks to the starting file in a loop. Now I know why Chris Duck did this in his script. Also spits out the error in the catch statement so it will at least tell you why it failed.

Cheers

[epedersen22]

Hooray – this version is working much better!

[a_user57]

Yup, awesome! Thanks very much!

[dsmelser93]

I tested on the ESXi with with a windows 2008 R2 virtual machine using *.vmdk files. The script runs without errors.

From within windows, everything looks great, lots of free space.

From Nimble, everything looks great, the volume size matches the use space reported by windows.

However, from vCenter shows a size warning on the datastore and when I browse the datastore, I see that the vmdk it at its maximum size. The virtual disk was thin provisioned, but now looks like a think provisioned virtual disk.

This does present a couple of challenges:

a) It affects how I provision datastores and virtual disks, because  vCenter warnings about datastores have been configured assuming a thin provisioned vmdk, but the script turns them to "thick" provisioned.

b) This is going to increase backup and recovery times. While the zeroed out space does compress well and won't take up space on the backup repositories (we use Veeam), it is increasing backup duration because all the zeros have to be read.

Does anyone have suggestions on how to best reduce the size of the vmdk file?

[Daniel-san]

On a VMDK, this is how you want to do it:

cd /vmfs/volumes

esxcli storage vmfs unmap -l VMFS-1

(VMFS-1 is the volume name)

Check out Nimble kb_customerPortal_KB-000065_Block_Reclamation.pdf article

[dsmelser93]

Can you provide a link to Nimble kb_customerPortal_KB-000065_Block_Reclamation.pdf

I can’t find it.