File Server iSCSI Volumes - In Guest vs RDM vs VMDK?

[mrbios48]

Hi, I'm trying to determine the best way of connecting my iSCSI volumes and struggling to find clear answers.

I'm currently using a 2012R2 File Server with 3 iSCSI Volumes consisting of 2.6TB, 2.5TB and 1.5TB. These are currently connected "The old fashioned way" aka in guest iSCSI.

However I have a few issues, the server has deduplication enabled and initially the space savings are reported back to the Nimble ok, but when either a backup or garbage collection (I'm unsure which is causing this) runs on the weekend the deduplication savings are lost to the Nimble but eventually return after a few days. Given the fact that the deduplication is unreliable in reporting space savings, it's not technically saving me any space. Would changing from in guest iSCSI to either RDM or VMDK produce any different or hopefully more reliable results from this aspect? What are the pros and cons of each?

Secondly in terms of backups, if i changed to a VMDK for example i could then make use of Arcserve UDP to backup and snapshot the data, but are VM snapshot backup softwares reliable with this quantity of data or would i be better off continuing to use Arcserve Backup for that aspect? I know there was some concern about that quantity of data back in vSphere 4 days which is why we opted to set it up with in guest iSCSI i think, but I'm now running vSphere 6.

Thanks

[jzygmunt70]

Windows deduplication moves a lot of stuff around while it is running the throughput optimization job.  It makes copies of blocks from one hidden dedupe file to another (in system volume information) and then deletes the original.  It is likely that the nimble sees theses blocks in use in both places because perhaps windows hasn't yet unmap'ed the deleted blocks yet, thus causing your volume to show it not saving any space until eventually windows unmap's the blocks and the nimble runs its internal garbage collection.  I'm not an expert on how windows 2012 does unmap, but it stands to reason that something like this could be going on.  In my experience, Windows dedupe is very inefficient and high maintenance when compared to dedupe appliances.  IMHO, It introduces a lot of hidden complexity at the file system level that you really should have a solid understanding of before using it if you want to be able to recover from a bad situation.

As far as the backup situation goes, tell us more about your design and what you're trying to do.  In general, running VMDKs instead of in guest iSCSI opens a lot of capability for backup for you.  With VMFS-5 which allows 62TB VMDKs, the only good reason I can think of for running in-guest iscsi for a file server is for doing cluster-across-boxes high-availability (I'm sure there's other good edge cases, but I can't think of any at the moment).  If you're not doing that, unless you have a really special use case, VMDKs are the way to go.

[kevin_hsieh]

Back in the vSphere 4 days, you could not run disks that large from VMDK. Now that you can, you are not getting any advantage from running via guest iSCSI. Converting to VMDK is probably a good idea.

Windows deduplication is going to cause havoc with the Nimble snapshots, because the blocks are going to keep changing. It will probably also mess up any backup that uses CBT, as it will see change rates much higher than what is real from a file system perspective.

[alex_goltz]

Jonathan and Kevin have excellent information here.  Your backup scenario has me/us intrigued.
Some time ago, I remember a post from either Glick's Gray Matter (or someone at Nimble) bragging about Windows De-dupe after Nimble compression.  TLDR; DON'T ENABLE Windows Server 2012 R2 de-dupe on Nimble volumes, unless you want an explosion of snapshot and cache problems!!!  Nimble doesn't support it anyways, so don't bother!  (Which bothers me why a Nimble blog would post it to begin with.)
I personally like direct iSCSI volumes because you then have the option for a cluster with little effort.  (But your backup software might dictate that decision.)
VMDK's are fine.  I just don't like that you have to "Punch zeros" with the VM off, in order to reclaim unused space on the VMDK file for that VM.  iSCSI volumes should just give it back to the Nimble, after you delete a large unused file from your file share.

Some notes from our past experiences:  Use the VMware paravirtual SCSI controller, and don't stack alot of VMDKs on the same SCSI chain.  (i.e. 0:0, 0:1, 0:2, 0:3, 0:4, 0:5).  When you add your VMDK disk, select a new chain (i.e. 1:0), which will create a new SCSI controller for you.

[lindy37]

"These are currently connected "The old fashioned way" aka in guest iSCSI."

I am fairly new to Nimble, but why not just leave it like that?  To me you have more flexibility and a more direct path to the volume/data with in guest iSCSI.  This is probably more useful when it comes to Exchange and SQL especially with synchronized snapshots.

As to the de-duplication on 2012 R2, we ran it at once but we had multiple issues with backup software, DPM no less at the time and we turned it off.

[chris24]

Hello,

I will second some very good information here and add mine:

1. Never use 2012 dedupe. You will save no space due to increase snapshot sizes and spend more time trying to recover space, besides it's a native feature coming to Nimble (unknown when).
2. Use VMDK's it's simple, less to manage and a lot easier to backup and you can run hypervisor based AV on it (tiny +).

Old school direct attached iscsi is great for applications such as Exchange / SQL to leverage VSS snapshots without those IO pauses you get in VMware. Stick to keeping it simple, manage everything through VMware.

[jtmoree23]

Original poster may not have this issue but we use ISCSI because our 2012 filers are in a cluster.  Clustering with VMDK is not well supported--nor is clustering well supported.  Vmware 6 has improved it a little. 

We had performance problems with quickbooks on vmdk.  (qb on vm is not supported and Intuit support hangs up on you if you tell them).

[jliu79]

For file server with large volumes I would not trust putting it in a huge VMDK/VHDX file, better to use in guest iscsi.

[noam126]

Any experience with Windows 2016 native dedup on Nimble volumes? We're using vmdk. At least for now we're not taking Nimble snapshots of the volume in question and not making CBT-enabled vm backups. It sounds like those concerns, at least, would also apply to dedup on Windows 2016. Likewise some cache-related performance penalty. Other concerns?