Array Setup and Networking
cancel
Showing results for 
Search instead for 
Did you mean: 

How big are your VMFS datastores/how many VMs?

 
SOLVED
Go to solution
jregel82
Occasional Contributor

How big are your VMFS datastores/how many VMs?

Just setting up our first Nimble and need to make some decisions about VMware VMFS datastore sizes and number of VMs per datastore.

Are there any recommendations from Nimble, and what are other users doing?

Thanks!

15 REPLIES
ndyer39
HPE Pro
Solution

Re: How big are your VMFS datastores/how many VMs?

Hi Julian,

Congratulations and thank you for your investment in Nimble Storage.

When this question comes up I tend to reference the following blog post from Jason Boche, who studied this subject and came out with a recommendation of 10 HIGH IO VMs, 15 AVERAGE IO VMs or 20 LOW IO VMs. You should stick to these limits too if you're planning on using vCenter quieced snapshots within Nimble, as the more VMs you have in a VMFS, the more VMs have to be sync'd and paused for the snapshot to take place.

jregel82
Occasional Contributor

Re: How big are your VMFS datastores/how many VMs?

Thanks for the advice Nick.

JR

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

It's not just the number of VM's but also the number of simultaneous VM snaps that can cause you issues. I believe that Vcentre can have 4 VM snaps running at any one time. So if you have a couple of "busy" VM's they can take longer to quiesce and ultimately lead to timeouts and those "failed to snap messages"

jeberhardt42
Occasional Advisor

Re: How big are your VMFS datastores/how many VMs?

Based on the Nimble and VMware training that I've gone through, best practices usually states that you should create a 1 - 1 - 1 relationship where you have 1 VM to 1 datastore to 1 SAN volume. This does increase the administration of your virtual network by having to monitor more volumes and datastores, but it does help with space conservation and provides more control of the VM environment. One big advantage to this configuration, on the subject of snapshots, is that if you needed to restore a snapshot from the SAN, you're ONLY restoring 1 virtual machine, not 10, 15, or 20 VMs. It would also help with allowing you to break apart the schedules for the VM snapshots that are being taken simultaneously, so vCenter is not timing out as stated by Mark.

Hope this helps!

Valdereth
Trusted Contributor

Re: How big are your VMFS datastores/how many VMs?

*cough*vvol*cough*  

Anyone happen to know what licensing level will be required for Virtual Volumes?  I'm assuming Enterprise Plus...

mgram128
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

I'm agreeing with Joe that the best performance and granular management can be achieved with the 1-1-1- provisioning plan.   However, keep in mind that ESX5.x hosts have a limit of 256 LUNs per server.     In practice, most deployments are a balance of multiple strategies.  

What I normally advise in the planning phase of your project is list out all of your business applications / processes that are going to be supported by the infrastructure.   Add to this the server instances that support those applications and processes.  Add to this what you believe will be a reasonable RPO and RTO and then go get buy in from the business that you are building this for.   Normally, its natural to see RPOs and RTO falling into three buckets that I will all A,B, & C where A apps have short (1 hourly or less RPOs and RTOs, B apps are medium ( 4-8 hr ) expectations, and then C apps are much longer ( Days to Weeks).

Then, with your A list, do the 1-1-1 provisioning, and for the most critical apps, do this for each volume on the server.   You would create a volume collection for each A group app and schedule the snaps to occur with no other apps +/- 15 min.   With your C and B group of apps, its much less critical and following the rules that Nick laid out (10 HIGH IO VMs, 15 AVERAGE IO VMs or 20 LOW IO VMs.) is pretty sage advice.

ccolht99
Advisor

Re: How big are your VMFS datastores/how many VMs?

  • VMware will not do more than 16 simultaneous snapshots with file system quiescence. Any more than that will only be crash consistent.  So you don't want to schedule more than that at a time. Your volume collections can't contain more than 16 vms each to stay within this limit.
  • If you go 1:1 you can't take advantage of SDRS or storage profiles.
  • You also can't have more than 256 luns per host. Assuming all hosts in a cluster see all luns, if you use a 1:1 ratio, you can't have more than 256 vms in a cluster. If you split OS/data vmdks on separate luns, your vm count goes down even more.
  • 1:1 wastes space since every datastore has overhead, even thin provisioned.

To manage IO, create separate datastore clusters and provision them as required. There are not many tuning dials on a Nimble. This is both a good thing and a bad one depending on what you want/need to do.

ndyer39
HPE Pro

Re: How big are your VMFS datastores/how many VMs?

Enterprise or Enterprise Plus I'm lead to believe...

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

A 1:1 ratio of VM's per datastore is impractical in the real world. If you are running hundred of VM's is carious VMware datacentres then you will very quickly consume all the array volumes 0-255. Not sure that 16 simultaneous snapshots is correct? Think it's 4 per VCentre server.

jeberhardt42
Occasional Advisor

Re: How big are your VMFS datastores/how many VMs?

The suggestion for the 1:1 ratio was based on the assumption that they were far underneath the 256 LUN maximum for a VM host. This configuration is not impractical in the real world by any means, it only depends on the size of the VM environment that you manage and license/options that you have to manage that environment. If you don't have enterprise or enterprise plus licensing, you can't take advantage of SDRS where compiling multiple VMs in a single datastore makes sense.

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

Fair point Joe.

                      Most environments I support are enterprise licence installations and have at least 100 VM's. Plus we use In-Guest iSCSI (and NCM) for SQL and Exchange so we'd soon run out of volumes on the array . As a general rule we have 10 SQL VM's per datastore and 15 max for general file and front end app servers. Only a guide and naturally one cap does not always fit all. The 1:1 ration mentioned give the same kind of advantages in terms of Snapshotting that In-Guest iSCSI does but with the advantage that VCentre isn't involved with the Quiescing process. I think the musc mooted VVols in VMware 6 will be a paradigm shift with a one to one relationship with the VM and it's VMDK as opposed o the datastre it resides on ......... If I've understood the web demos correctly. Matbe someone will enlighten me in this regard. Again it appears that Enterprise or even only Enterprise plus will have this feature and that's seems rather mean.

Regards,

              Mark.

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

Excuse the Human to Keyboard errors in previous comments. Doh!

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

SDRS
is only available with Enterprise Plus...... so annoying.

Valdereth
Trusted Contributor

Re: How big are your VMFS datastores/how many VMs?

marktheblue45
Valued Contributor

Re: How big are your VMFS datastores/how many VMs?

Hi Chuck,

               I have haven't seen more than 4 simultaneous snapshots. What happens is that 4 run and when one or more of the 4 finish others begin the snapshot process. If some VM's are busy and take a long time to quiesce it delays the whole process and exceeds the default Nimble timeout setting. I have use Nimble VMware integrated snapshots on Datastores with 20 VM's with no problem but they were "quiet" VM's. As a rule I have a max of 15 VM's per protected Volume collection.

Kind Regards,

                      Mark,