How big are your VMFS datastores/how many VMs?

[jregel82]

Just setting up our first Nimble and need to make some decisions about VMware VMFS datastore sizes and number of VMs per datastore.

Are there any recommendations from Nimble, and what are other users doing?

Thanks!

[Nick_Dyer · Accepted solution]

Hi Julian,

Congratulations and thank you for your investment in Nimble Storage.

When this question comes up I tend to reference the following blog post from Jason Boche, who studied this subject and came out with a recommendation of 10 HIGH IO VMs, 15 AVERAGE IO VMs or 20 LOW IO VMs. You should stick to these limits too if you're planning on using vCenter quieced snapshots within Nimble, as the more VMs you have in a VMFS, the more VMs have to be sync'd and paused for the snapshot to take place.

[jregel82]

Thanks for the advice Nick.

JR

[marktheblue45]

It's not just the number of VM's but also the number of simultaneous VM snaps that can cause you issues. I believe that Vcentre can have 4 VM snaps running at any one time. So if you have a couple of "busy" VM's they can take longer to quiesce and ultimately lead to timeouts and those "failed to snap messages"

[jeberhardt42]

Based on the Nimble and VMware training that I've gone through, best practices usually states that you should create a 1 - 1 - 1 relationship where you have 1 VM to 1 datastore to 1 SAN volume. This does increase the administration of your virtual network by having to monitor more volumes and datastores, but it does help with space conservation and provides more control of the VM environment. One big advantage to this configuration, on the subject of snapshots, is that if you needed to restore a snapshot from the SAN, you're ONLY restoring 1 virtual machine, not 10, 15, or 20 VMs. It would also help with allowing you to break apart the schedules for the VM snapshots that are being taken simultaneously, so vCenter is not timing out as stated by Mark.

Hope this helps!

[Valdereth]

*cough*vvol*cough*  

Anyone happen to know what licensing level will be required for Virtual Volumes?  I'm assuming Enterprise Plus...

[mgram128]

I'm agreeing with Joe that the best performance and granular management can be achieved with the 1-1-1- provisioning plan.   However, keep in mind that ESX5.x hosts have a limit of 256 LUNs per server.     In practice, most deployments are a balance of multiple strategies.  

What I normally advise in the planning phase of your project is list out all of your business applications / processes that are going to be supported by the infrastructure.   Add to this the server instances that support those applications and processes.  Add to this what you believe will be a reasonable RPO and RTO and then go get buy in from the business that you are building this for.   Normally, its natural to see RPOs and RTO falling into three buckets that I will all A,B, & C where A apps have short (1 hourly or less RPOs and RTOs, B apps are medium ( 4-8 hr ) expectations, and then C apps are much longer ( Days to Weeks).

Then, with your A list, do the 1-1-1 provisioning, and for the most critical apps, do this for each volume on the server.   You would create a volume collection for each A group app and schedule the snaps to occur with no other apps +/- 15 min.   With your C and B group of apps, its much less critical and following the rules that Nick laid out (10 HIGH IO VMs, 15 AVERAGE IO VMs or 20 LOW IO VMs.) is pretty sage advice.

[ccolht99]

• VMware will not do more than 16 simultaneous snapshots with file system quiescence. Any more than that will only be crash consistent.  So you don't want to schedule more than that at a time. Your volume collections can't contain more than 16 vms each to stay within this limit.
• If you go 1:1 you can't take advantage of SDRS or storage profiles.
• You also can't have more than 256 luns per host. Assuming all hosts in a cluster see all luns, if you use a 1:1 ratio, you can't have more than 256 vms in a cluster. If you split OS/data vmdks on separate luns, your vm count goes down even more.
• 1:1 wastes space since every datastore has overhead, even thin provisioned.

To manage IO, create separate datastore clusters and provision them as required. There are not many tuning dials on a Nimble. This is both a good thing and a bad one depending on what you want/need to do.

[Nick_Dyer]

Enterprise or Enterprise Plus I'm lead to believe...

[marktheblue45]

A 1:1 ratio of VM's per datastore is impractical in the real world. If you are running hundred of VM's is carious VMware datacentres then you will very quickly consume all the array volumes 0-255. Not sure that 16 simultaneous snapshots is correct? Think it's 4 per VCentre server.