Jumbo frames end-to-end? Or not?

[sbaldridge16]

Seeking advice on this one.  I have jumbo enabled on my CS240, the switches, and my storage stack on the ESXi 5.1 hosts/vDS are all using jumbo.  I created a new port group for iSCSI VM traffic and it will use jumbo.

Here is my question:  I have added two nics to a VM and configured them for iSCSI MPIO but I don't know if the Win2008R2 VM will need to use jumbo or not?  I searched here and didn't see an answer.

Opinions?

Thanks in advance.

Scott

[bgrieve65 · Accepted solution]

Hi Scott,

You should enable jumbo frames across the entire network stack (host included) to realize the benefit that jumbo frames can offer. You can enable/use jumbo frames in Windows 2008 and later, and you will need to enable it on the network adapter(s) themselves. Here is a link to a basic article that outlines the steps involved:

Q. How do I enable jumbo frames? | Windows content from Windows IT Pro

For what it's worth, I would also submit that jumbo frames does not *always* equate to better performance. In most of the customer environments I'm exposed to, jumbo frames seems to be more common (and beneficial) where 10Gb Ethernet is in use and much less used in 1Gb environments.

Hope this helps!

Brandon Grieve

Nimble Storage

[sbaldridge16]

Thank you.  That makes sense.

[jliu79]

Hi Brandon, can you explain why 10GB network gets more benefit out of jumbo frames? Thanks.

[bgrieve65]

Hello Jason. It generally has to do with the fact that 10G networks often carry larger data loads. So, larger frames means less transmissions need to take place on the network, thereby reducing overhead.

[julez66]

I'd like to see a couple benchmark comparisons of Jumbo frames vs standard 1500 MTU on Nimble.

[marktheblue45]

We have two 460G arrays but the ESX hosts have two physical 1GB NICs per VM network. Without any formal before and after benchmarks I'd suggest Jumbo frames on the iSCSI network at very leas will improve performance since all roads lead to Rome as far as iSCSI traffic is concerned. Fewer transmissions/packets albeit larger ones. Not convinced whether general 1GB LAN traffic will benefit at all but the 1GB Switches have a 20GB Backplane so maybe still worth considering!

[marktheblue45]

The cost of 10GB cards for the hosts plus a concentrator or two rack up the costs!

[marktheblue45]

The point I'm trying to make is that if the storage has 10GB interfaces on iSCSI and you have a number of hosts connecting on the iSCSI network then it's many to one (dual port on Nimble) so definitely I'll recommend Jumbo frames for iSCSI traffic.

[apolman45]

Julez,

I've done some testing in the past with jumbo vs standard, and found there's little (couple of percent) performance gain from using jumbo frames. Personally I recommend against jumbo frames for reasons of simplicity. I have done installs at 2 big environments where we found out that not all of the layers (a NIC in one environment, a switch port in another) were configured for jumbo frames. This resulted in lower then expected performance as you can imagine.

Keep It Simple!

Arne

[Nick_Dyer]

I have to +1 Arne Polman here. Unless you're looking to absolutely stress a 10GbE network with major sequential IO, Jumbo Frames will give little performance benefit vs the relative management & support overhead it takes to administer and troubleshoot the environment when it goes wrong - reason being it acts like a "house of cards"; one misconfiguration in the environment and the whole thing falls down.

Jumbo Frames should only really be considered on 10GbE networks (1GbE rarely sees any improvement), and only after the environment has been run at 1500 MTU to check the performance and latency first. In fact Support's best practice is to install & configure on 1500 MTU, and then look at switching to Jumbos only if there may be a good case to do so.

I think of KISS - Keep It Simple, Stupid

[alex_goltz]

Add me to the list of jumbo frames on iSCSI Initiators on 10Gbit.  Our heaviest workloads are SQL 2012 sequential reads.  At times our data warehouse VM will get close to 2 GB/sec if the cache hit ratio is near 100%.

This is probably the only scenario we have that gets the more mileage out of the additional payload size.

I would agree with Nick Dyer.  Test 1500 first, then move to 9000 if you really feel the calling to do so.

To whomever is interested:

One command I like to run on the guest VM to make sure the jumbo frame settings are correct in all areas is:  ping <array iSCSI IP> -f -l 8972                Don't use the iSCSI discovery IP. Use one of the actual data IPs.  And use a lowercase "L" for the second switch.

[davecramp16]

I find it's best to explain Jumbo Frames as a way to reduce workload of switches and network adapters rather than something that's really going to increase performance.

Increasing the packet size can reduce the number of packets required to send traffic, which reduces the workload of switches and other network equipment (adapters etc..) especially in a TCP world where the equipment runs checksums and other sequence checking on every packet.

This is a lot more visible on 10Gb networks where the number of packets per second will typically be much higher than that of a 1Gbit network.