Question on nimble performance policy for ESX and block size tuning

[arutledge91]

It my understanding from best practice doc(bpg_nimble_storage_vmware_vsphere5.pdf) that any LUN which will contain VMFS datastores should be set to use the ESX performance policy.

It is also my understanding that this policy uses a 4k block size on nimble.

My question is in regards to the block alignment document: https://connect.nimblestorage.com/docs/DOC-1185.  Does the table on page 3/4 also apply to VMDKs on VMFS volumes or does this only apply to volumes directly mounted using MS iSCSI initiator, etc?  If it does apply to VMFS should there be a column referencing VMware block size of 1MB(or 8k sub allocation) blocks, etc and that taken into account when discussing?

Thanks

[Nick_Dyer]

Hi Alan,

Block alignment only really needs to be a concern if you're using VMFS volumes which were created using VMFS-3, or if you're using Windows 2003 (or a volume which was created on Windows 2003 and upgraded to Windows 2008). If it's VMFS-5 or a new volume created on Windows 2008 or later, then it *should* be properly aligned.

If you're using VMware vSphere 5, please use "VMware ESX 5" as the performance policy rather than the "VMware ESX" version; this means that the latest changes to the limitations of VMFS-5 are able to be used (ie VMFS > 2TB in size).

Support have a great tool called Coral (which is due to be integrated into the next rollout of Infosight) which allows for diving into alignment and latency of a volume to spot the problem. If you give Support a call or email they can give you this information if you have concerns.

[arutledge91]

Nick,

Thank you for the response but my question is in regards to block size tuning for performance and not alignment.

[alex_goltz · Accepted solution]

Hi Alan,

I think you're on the right track with that table on 3/4 with that PDF.  For whatever its worth, I referenced that document when I was building our SQL servers with direct iSCSI to the Nimble volume.

Personally, I think the document should be Block Alignment AND Sizes for Best Performance on Nimble Storage.  That might alleviate some confusion.

This document is alot more helpful when configuring iSCSI direct connections and NTFS cluster sizes, than block alignment.  Hopefully that table makes it into the documentation, because I think alot of people might be setting it up incorrectly until they come across this PDF.

You won't need this table if you are doing VMDK/VMFS datastores or RDMs.  (Those are 4KB I/O's with 1MB blocks on the storage, like you stated.  I think.)

Does this help ?

[arutledge91]

Alex,

Yes that absolutely answers my question, thanks. 

To dig deeper, from what I am reading with VMFS5 they are using 8k sub allocation blocks when writing files smaller than 1MB(sources below) do you think it is worth trying 8k block size policy instead of 4k for VMFS5 datastores?  For example, formatting NTFS cluster size at 8k, and setting nimble at 8k for VMFS datastore LUNs?  Seems like that would be highly optimized for file sizes above 1k and below 1MB.

http://www.pearsonitcertification.com/articles/article.aspx?p=1941024&seqNum=4

http://blogs.vmware.com/vsphere/2012/02/something-i-didnt-known-about-vmfs-sub-blocks.html

[boerlowie42]

Keep in mind that almost all of your IO on a VMFS Datastore will come from inside the Virtual Machine (which uses the blocksize from the OS/app inside the VM).

So the blocksize of a VMFS volume (which is now 1 MB with indeed sub-blocks) is important for all the files on a VMFS datastore, like the vmx file, logfiles, but doesn't really matter much for IOs in your VM.

Good example.  I have a VM which has Exchange installed.  All disks in the VMs are stored as VMDK files on a VMFS-5 datastore.  The C:\ drive/VMDK is on a datastore set with a Performance Policy of 'ESX v5' on the Nimble array (= 4KB).  The Exchange Database is on a seperate VMDK in a dedicated VMFS datastore for Exchange.  This has a Performance Policy of "Exchange 2010" on Nimble (= 32 KB).  This setup was validated by Nimble support as well.

The reason i believe for ESX having a 4 KB block size on Nimble is because of the random/mixed nature of the VMs on those datastores.  Most of the IOPS are 4 KB and if you have specifics, you should try to seperate them.

[arutledge91]

Thank you Sammy, that is extremely helpful and makes perfect sense.