HPE Storage Tech Insiders
cancel
Showing results for 
Search instead for 
Did you mean: 

Adventures in Nimble Storage and Windows 2012 R2 Deduplication

gary_martin

I have been having some adventures in Windows 2012 R2 Deduplication and thought I would share my experiences here for others who might be working on similar migrations.  As a long term NetApp user it's been interesting getting to grips with something different.  The biggest change for me is the loss of the NAS functionality which means I need something to do that NAS role in Nimble world.  Step up Windows 2012 R2 and more importantly the deduplication feature.  Note that a lot of the information here is not unique to a Nimble experience.


So, background first

 

We are still in the process of migrating to Nimble, although now a significant portion of the data has been moved (thanks Storage vMotion).  The largest dataset was our file shares (previously serviced by the NetApp storage) to a new Windows 2012 R2 file server hosted on Nimble.  We are not using much in guest iSCSI as we have adopted Veeam as our backup tools, so it is much easier to service those disks as VMWare disks rather than direct attached Nimble disks (via iSCSI).  An area of weakness in my mind is the ability to get data from Nimble to tape etc.  Nimble to Nimble is incredibly easy and for those brave enough to ditch tape it's snapshotting is both efficient simple to configure.  For my environment Veeam was the most applicable replacement for NetApp's data protection products (SnapDrive, SnapManager, Snapmirror and Snapvault).  We moved from Snapmirror to tape to Veeam and repositories.  My hope is that Nimble improves in a few areas (specifically authentication and SnapDrive like features in NCM), but we are where we are and the performance of these boxes vastly outweighs any problems I have with the software stack.  I'm sure this stuff will come and when it does we will really be cooking on gas.


On to my migration.

 

The Process

 

I moved each Netapp file share to a new disk on a Windows 2012 R2 file server I created.  Each share was given a new disk (I'm still on ESX 5.1 so maximum disk size is limited to 2TB).  There was a lot of data (nearly 10TB), but I seeded it using Robocopy then did the final sync and the rest of the magic on a weekend.  Also as part of the migration I enabled Microsoft Deduplication feature.  I thought as Nimble didn’t have this feature it would be worth a go and we did see some significant savings.

 

Currently – Used disk space is 3.3TB

Saved – 5.4TB

Unoptimised – 9.6TB

 

Those are pretty good numbers.  Unfortunately they are not reflected in the Nimble used space.  Nimble reports 6.8TB used including savings for compression.  It's not the 9.6TB of RAW data, but also not the tidy 3.3TB Windows thinks it is using.

 

Remediation

 

It seems pretty clear that we are dealing with UNMAP here.  VMWare doesn’t support guest UNMAP (won’t pass through UNMAP requests directly to the storage from the guest) so the dirty blocks (pre-Deduplication) are being marked as used storage.  I’ve seen some information about ESX 6 having some capabilities for OS support UNMAP but I think they are being very careful because some arrays (not Nimble’s) can be saturated by such requests.  This left me a bit stuck as there is ~3TB out there being used which is effectively deletions (pre-Windows dedupe data).  I did some digging on the Nimble Connect forums as found some information regarding reclaim (sdelete and a powershell alternative).  It was very focused on the layout where Nimble disks are direct attached to the guest (or physical machine), but I thought I would give the reclaimer a quick spin on one of the smaller (but well serviced by deduplication) disks.

 

I believe there are some caveats here.

1.       No snapshots should be enabled on the Nimble volume (you will just end up with a big snapshot)

2.       No VMWare snapshots should exist on the machine (same as above)

 

The space is coming back to me now as I guess the dirty blocks are being zeroed and Nimble is compressing it (lots of zeros compressed is probably a couple of zeros).  My hope is that I can get below 3.3TB used on the Nimble (3.3TB is the currently used disk space reported by Windows, so compression should be able to reduce this further).  Obviously I am running this as a one off operation so things can get dirty again from now on, but it should give me a clean base and a better correlation between OS used, Hypervisor used and Nimble used.


I finally managed to get down to 3.46TB used, pretty close to my 3.3TB target (I needed to add OS really so 3.3TB was optimistic).  There is something to note here though and that is that the compression for the volume holding all of this data is 1.04x.  That ratio is poor and I think Windows 2012 R2 Deduplication is a lie, it certainly looks like they are compressing this stuff too (not just deduplication).  Still, the results are good and far beyond what NetApp Deduplication gave me (although I didn't use compression there).


What if I used iSCSI in the Guest?


I believe, and might need correcting if I am wrong, that if I had the disks directly attached to the guest that Nimble would leverage the OS UNMAP and TRIM functionality to recover the space from the volume.  Windows 2012 supports these features and I would expect different results.  If and when Nimble's data protection features get to a point where this is feasible for me I might change how the file server is architected.


Lessons For Others


I think most of the info can be gleaned from above, however I did make an unrelated balls up.  When I first did my Robocopy I didn't use the /SEC switch so the permissions didn't carry across.  I ended up in a position where all the files had default file system permissions, but the directories had the correct permissions.  If I picked this up right away I would have run Robocopy with /SECFIX but I didn't.  I'm kicking myself for missing it and it's caused a few issues for the users accessing the data (although really highlights that I probably migrated years of data that no-one really cares about as it has taken nearly a month for issues to show up).  Here is a handy powershell script that takes the folder security and applies to files underneath (this enables inheritance on the files only for those who say I could have just gone to folder properties and replaced file permissions on all child objects).  I went through several revisions (initially calling icacls.exe /reset for each file) but this is the fastest I could make it.  It might help a googler trying to fix the same issue.


# Recurse($path)

 

function Recurse ([string]$path){

  if (-not (Test-Path $path)) {

 

   Write-Error "$path is an invalid path."

   return $false

 

  $path

  $inheritance = get-acl $path

  $inheritance.SetAccessRuleProtection($false,$false)

  Get-ChildItem -Path $path -Attributes !Directory | set-acl -aclobject $inheritance

  $directories = Get-ChildItem -Path $path -Attributes Directory  | Select-Object Fullname

  foreach ($directory in $directories) {

   Recurse $directory.FullName

  }

}

 

About the Author

gary_martin

Comments
jpmeredith

Nice write-up. One thing to note -- a big tradeoff in changing to in-guest iSCSI volume mapping is you lose your Veeam backup capability (unless they've changed something recently), so you'd be back to square one on your tape backup strategy. About the only option I'm aware of is Commvault, which is the only thing that offers any integration with Nimble's snapshots, or doing in-guest backups via agent or something like that.

I love how Nimble's snapshots work for user files when NCM is doing the shadow copies, but I I was hoping Nimble would be farther along by now integrating with some of the big players like Veeam for customers who can't (or won't) bite of a big chunk of Commvault. From what I hear it's a big expensive (though capable) piece of software.

gary_martin

It really depends on where Nimble go with their data protection software.  Automation at the moment is a pig, especially having to reveal the authentication string in any dirty scripting to get snapshots mounted to servers to backup the data to tape.  This is where I am looking for some of that nice Nimble innovation.  Haven't got a big enough wallet for CommVault.

jpmeredith

Yep, getting clones mounted to a utility server to run the backups is something I've been monkeying with lately, but it's clunky (especially since I suck at everything scripting).

Valdereth

Thank you for sharing your experience!

Since its just a Windows File Server, maybe Veeam Endpoint could be used to catch the in-guest iSCSI attached volumes?  You'd want to look through pros/cons but it might be an option depending on if dedupe and unmap is needed.

What did your Windows Dedupe settings look like?

I remember hearing that it's best to turn off compression on guest deduped volumes, here's a thread you might have already viewed with some thoughts/recommendations: Re: dedupe savings with 2012

gary_martin

I could use Veeam Endpoint, but at some point I might be replicating the machine.  Might be easier to do that as a VM, rather than VM with external disks (direct attached).

Deduplicaton settings pretty standard, turned on for disk and set schedule for out of hours.  Do anything over 3 days old.

I may turn off compression, although it will still be compressing the OS disk.

Nimble scripting for backup for direct attached disks to tape.  Here's my effort to achieve this.  This uses a lot from others (thread referenced below). 

#  Script to create snapshot of a volume and then attach locally.  Break out the script to attach the disk to another
#  machine for backup.
#  Most credit goes to the guys contributing to this thread https://connect.nimblestorage.com/message/4501#4501


#This bit contains all the variables you need to supply these
#Your Array Name and login details
$NimbleArray = "MyNimbleArrayName"
$NimbleLogin = "admin"
$NimblePassword = "admin"

#Snapshot retention length

$SnapshotKeepDays = 14
#Basename for snapshot (will include date when created as snapshot)
$SnapshotBaseName = "MySnapshotName-"
#The volume you want to backup
$SnapshotVolume = "VolumeToBackup"

#The Initiator group to add to the backup volume

$BackupServerInitiatorGroupName = "NimbleInitiatorNameForBackupServer"

#This bit builds some variables to use later.  You can change bits if you want.
#The name for the clone you will create, this is built from parent volume name

$SnapshotVolumeClone = $SnapshotVolume + "-Backup"
#This bit builds the target name for you.
$VolumeInitiatorTargetName = "iqn.2007-11.com.nimblestorage:" + $SnapshotVolumeClone + "*"


#Connect to the Nimble
Connect-NSArray -SystemName $NimbleArray -UserName $NimbleLogin -Password $NimblePassword


#Create a new Snapshot daily

$SnapDate = Date | Out-String
$SnapshotName = $SnapshotBaseName + $SnapDate.Trim()
$SnapshotName = $SnapshotName -replace " ","-"
$SnapshotName = $SnapshotName -replace ":","-"
Get-NSVolume $SnapshotVolume | New-NSSnapshot -Name $SnapshotName

#Delete snaphots that exceed $SnapshotKeepDays.  Note if you are creating other snapshots you will need to filter the bit | Get-NSSnapShot.

$VolSnaps = Get-NSVolume $SnapshotVolume | Get-NSSnapShot | Sort-Object LocalCreationTime -Descending
$Counter = 0
Foreach ($Snap in $VolSnaps) {
$Counter = $Counter + 1
If ($counter -gt $SnapshotKeepDays) {
Remove-NSSnapShot $Snap -Force
}
}

#You can port this bit to another server if you want to disk to be mounted there.  Remove the date part if you do,
#as the scheduler on that individual script element can be used.

#If it is a Friday then mount the latest snapshot
If ((get-date).DayOfWeek -eq "Friday") {

#Get rid of the old connected Snapshot.
Remove-NSVolume -Name $SnapshotVolumeClone -force
"rescan" | diskpart

#Do the Nimble side of presenting the disk

Get-NSVolume $SnapshotVolume | Get-NSSnapShot | Sort-Object LocalCreationTime -Descending | Select-Object -First 1 | New-NSClone -Name $SnapshotVolumeClone
Add-NSInitiatorGroupToVolume -InitiatorGroup $BackupServerInitiatorGroupName -Volume $SnapshotVolumeClone
Update-IscsiTarget
Get-IscsiTarget -NodeAddress $VolumeInitiatorTargetName | Connect-IscsiTarget

"rescan" | diskpart

#This code was borrowed from https://connect.nimblestorage.com/message/4501#4501.  Get the clone disk available.

### get Nimble disks ###

$nimbledisks = get-disk | Where {$_.Manufacturer -like "Nimble*"} | select -expandproperty "number"

### Remove readonly ###

write-host "remove readonly atribute from disks > $nimbledisks" -foregroundcolor yellow

foreach( $disk in $nimbledisks ){ set-disk -number $disk -isreadonly 0}

write-host "Disks > $nimbledisks -> writeable" -foregroundcolor green

### Remove offline ###

write-host "remove offline atribute from disks > $nimbledisks"  -foregroundcolor yellow

foreach( $disk in $nimbledisks ){ set-disk -number $disk -isoffline 0}

write-host "Disks > $nimbledisks -> online" -foregroundcolor green
}

mrbios48

Thank you for this write up Gary. I'm in the process of doing a very similar file server move myself, with the only difference being that i'm using in guest iSCSI, and i hadn't even considered the additional functionality of de-duplication until i read this post.

I'm currently doing the initial robocopy seed from a 2008R2 file server using our old storage setup to a new 2012R2 file server using our shiney new nimble storage. I'll be enabling de-duplication and i'll be sure to report back as to whether in guest iSCSI functions differently to what you're seeing.

gary_martin

Hey James, will be interesting to know if UMAP returns space to your array after the deduplication has run.  Server should be able to indicate directly to the array when the block is finished with so as space savings goes up on the host, used space on the Nimble goes down. 

As I am using vmdk, within a vmfs, within a Nimble volume there is no way for host to indicate the block is finished with.

How are you managing RPO?  Are you going to have snapshots for some time on the array or attach the disk to a backup server for traditional backup?

Keep in mind that if you do an array based snapshot before initial deduplication has completed you will not see space return to the volume until the pre-deduplication snapshot is removed.

mrbios48

Hi Gary,

I've made the move now, and as you suspected in your blog i am indeed seeing the space savings return to me on the Nimble using in guest iSCSI connections. I sent the data over via robocopy over night, enabled de-duplication the next day and over the next night i had a steady decrease in storage use on my Nimble graphs.

In terms of backups I'm using a combination of Arcserve UDP for my VMs and Arcserve Backup for the more traditional data backup. I do a D2D incremental backup routine that runs daily to slow storage in arcserve backup for the in guest iSCSI mounted data, which then backs up to tape weekly, and also backs up my UDP snapshots to tape on a monthly basis. I opted not to use the Nimble snapshot functionality, mainly due to what i already had in place prior to purchasing the Nimble, rather than any technical reason.

I've changed my tape backups to point directly at the file server rather than the slower D2D store now, and enabled a setting in arcserve backup that suggests some sort of de-dupe intelligence. I'm unsure whether that's actually saving my space on the tapes yet though, i need to rethink that aspect a bit.

Events
Apr 24 - 25, 2018
Online
Expert Days - 2018
Visit this forum and get the schedules for online HPE Expert Days where you can talk to HPE product experts, R&D and support team members and get answ...
Read more
June 19 - 21
Las Vegas, NV
HPE Discover 2018 Las Vegas
Visit this forum and learn about all things Discover 2018 in Las Vegas, Nevada, June 19 - 21, 2018.
Read more
View all