Number of VMs per Datastore

[jrich52352]

hmm that very well could be it, i'll give that a try tomorrow, thanks

[boerlowie42]

While this was indeed true in the past, locking is handled by the Nimble on block level by VAAI ATS.  10 VMs per datastore is more like ESX 3.5 guideline.  We go way above that on  vSphere 5 (on different kinds of storage but all with VAAI ATS enabled) and don't see those issues...

Find it strange that Nimble recommend such a low number per datastore?  I mean if you use VDI on nimble with linked clones, most will go with more than 10 VMs per datastore.

Interesting to see how this thread will develop :-)

[tjoffs71]

Sammy,

In VDI, you are looking at a whole different beast.  The issue is more apparent with NON-VDI work loads (especially without VAAI) due to Queue Depth, The Number of Hosts, and VM Per Host, Per Data Store Saturation Levels.  In every scenario, you need to do the math, check the Queue Depth ratio, the I/Ops requirements, read/write profile (look at 50/50 for Linked Clone Datastore, 80/10 for persistent Disk, and 100% Read for Replica Datastore -- If you mix these it gets more interesting).  Ultimately with storage, I/Ops are great but if you can't request them -- well then they are just a feature set that does not matter.  When dealing with iSCSI storage especially, you must look at the whole picture (I/Ops. Queue, Network (Pipe, Jumbo Frames, QoS, Etc.), Host Capabilities, Etc.).

I think the key take away from this whole discussion is perhaps two points:

1) Everyone's scenario is different, and vendors will normally suggest industry standard practices as a rule.

2) When designing storage, look at the whole picture, do the math and put in what works best for your needs.

[boerlowie42]

Completely agree Ted,

i was pointing more to the locking issue they were referring to.  With VAAI ATS, i don't see that as an issue with 10+ VMs.  And when doing snapshots etc. you are doing quite some locks.

It indeed all depends a bit on the situation and environment!

[jrich52352]

So based on all of these discussions, I dont see any reason i'd have an issue here.

Also i checked the vcenter host this morning it was 4cpu with 8GB, i've since upped that to 16.

there are two data stores on the env with 8 hsots.

4 hosts are in a cluster with one data store (due to MS lic) this is considered core services. vcenter, DC's, other such utility type boxes. there is a total of 19 VMs on this cluster (no snapshots)

The other side is again 4 hosts, with a total of 39VMs, of those 28 of them are on the datastore im trying to snapshot and the rest are on the datastore from above.

the 28 hosts on datastore im trying to snapshot is to be used like VDI. We are building up the dev env based on production level code with a scale back of number of nodes. This is being used by no one currently and the snapshots are schedules to get VSS versions to be used for the clones for the developers. this base env (datastore) will never have users on it.

so in all of these questions there is nothing that says to me this should fail.

plenty of power with the hosts (HP chassis system, C7000 with beefy hosts)

almost all VMs are idle since im building all of it from scratch and there are no users of any kind.

Looking at the perf data i am seeing a spike in utilization of that volume between 12am-2, but the spike is 3.6K iops and 60MBs.

I'll dig in to see what this spike is from but even these numbers dont seem nuts compared to what this box should be able to do.

[bleblanc15]

Hey Justin,

Do you need to quiesce/snapshot the VMs? Seems like crash consistent, array based, snapshots would be good enough in this situation. Then, it wouldn't matter how many VMs you had in the Datastore.  Especially if you have broken out the DB volumes from the OS volumes.

[jrich52352]

for the time being the DBs are on the same datastore, which is why i was trying to use the nimble protection policy to generate the snapshots so that they would be crash consistent.

from another post what i found was that the only way to trigger VSS was to do a protection policy

[jrich52352]

So the memory addition made no difference.

Im trying to track down events within vmware to indicate what caused the problem but so far no luck. anyone know where i might find such events? with powercli im poking around with the get-vievent but they dont seem to have much for details.

[boerlowie42]

Take a look on the 'Tasks & Events' tab of the VM to see if the 'Create snapshot' task succeeds.  All that Nimble does is send a Create Snapshot command to vCenter.

If it fails, you can look in the Event Log of the VM to see the actual VSS error.

[jrich52352]

I dont see any failures, but some of them dont have snapshots on them. Im tried taking snapshots of them and they all go successfully. I suspect they only do one at a time and its not finishing in a timely manner. The tech said he can extend the timeout period (default 10min) to something longer.

I've felt i've had other performance issues so I might work with a tech to try and tack that down.

I still feel that this unit should be able to execute the snapshots in under 10 min.