Array Setup and Networking
cancel
Showing results for 
Search instead for 
Did you mean: 

Jumbo frames end-to-end? Or not?

SOLVED
Go to solution
sbaldridge16
Occasional Contributor

Jumbo frames end-to-end? Or not?

Seeking advice on this one.  I have jumbo enabled on my CS240, the switches, and my storage stack on the ESXi 5.1 hosts/vDS are all using jumbo.  I created a new port group for iSCSI VM traffic and it will use jumbo.

Here is my question:  I have added two nics to a VM and configured them for iSCSI MPIO but I don't know if the Win2008R2 VM will need to use jumbo or not?  I searched here and didn't see an answer.

Opinions?

Thanks in advance.

Scott

12 REPLIES
bgrieve65
Valued Contributor
Solution

Re: Jumbo frames end-to-end? Or not?

Hi Scott,

You should enable jumbo frames across the entire network stack (host included) to realize the benefit that jumbo frames can offer. You can enable/use jumbo frames in Windows 2008 and later, and you will need to enable it on the network adapter(s) themselves. Here is a link to a basic article that outlines the steps involved:

Q. How do I enable jumbo frames? | Windows content from Windows IT Pro

For what it's worth, I would also submit that jumbo frames does not *always* equate to better performance. In most of the customer environments I'm exposed to, jumbo frames seems to be more common (and beneficial) where 10Gb Ethernet is in use and much less used in 1Gb environments.

Hope this helps!

Brandon Grieve

Nimble Storage

sbaldridge16
Occasional Contributor

Re: Jumbo frames end-to-end? Or not?

Thank you.  That makes sense.

jliu79
Frequent Advisor

Re: Jumbo frames end-to-end? Or not?

Hi Brandon, can you explain why 10GB network gets more benefit out of jumbo frames? Thanks.

bgrieve65
Valued Contributor

Re: Jumbo frames end-to-end?  Or not?

Hello Jason. It generally has to do with the fact that 10G networks often carry larger data loads. So, larger frames means less transmissions need to take place on the network, thereby reducing overhead.

julez66
Frequent Advisor

Re: Jumbo frames end-to-end? Or not?

I'd like to see a couple benchmark comparisons of Jumbo frames vs standard 1500 MTU on Nimble.

marktheblue45
Valued Contributor

Re: Jumbo frames end-to-end?  Or not?

We have two 460G arrays but the ESX hosts have two physical 1GB NICs per VM network. Without any formal before and after benchmarks I'd suggest Jumbo frames on the iSCSI network at very leas will improve performance since all roads lead to Rome as far as iSCSI traffic is concerned. Fewer transmissions/packets albeit larger ones. Not convinced whether general 1GB LAN traffic will benefit at all but the 1GB Switches have a 20GB Backplane so maybe still worth considering!

marktheblue45
Valued Contributor

Re: Jumbo frames end-to-end?  Or not?

The cost of 10GB cards for the hosts plus a concentrator or two rack up the costs!

marktheblue45
Valued Contributor

Re: Jumbo frames end-to-end?  Or not?

The point I'm trying to make is that if the storage has 10GB interfaces on iSCSI and you have a number of hosts connecting on the iSCSI network then it's many to one (dual port on Nimble) so definitely I'll recommend Jumbo frames for iSCSI traffic.

apolman45
Advisor

Re: Jumbo frames end-to-end? Or not?

Julez,


I've done some testing in the past with jumbo vs standard, and found there's little (couple of percent) performance gain from using jumbo frames. Personally I recommend against jumbo frames for reasons of simplicity. I have done installs at 2 big environments where we found out that not all of the layers (a NIC in one environment, a switch port in another) were configured for jumbo frames. This resulted in lower then expected performance as you can imagine.

Keep It Simple!


Arne

ndyer39
HPE Pro

Re: Jumbo frames end-to-end? Or not?

I have to +1 Arne Polman here. Unless you're looking to absolutely stress a 10GbE network with major sequential IO, Jumbo Frames will give little performance benefit vs the relative management & support overhead it takes to administer and troubleshoot the environment when it goes wrong - reason being it acts like a "house of cards"; one misconfiguration in the environment and the whole thing falls down.

Jumbo Frames should only really be considered on 10GbE networks (1GbE rarely sees any improvement), and only after the environment has been run at 1500 MTU to check the performance and latency first. In fact Support's best practice is to install & configure on 1500 MTU, and then look at switching to Jumbos only if there may be a good case to do so.

I think of KISS - Keep It Simple, Stupid

alex_goltz
Advisor

Re: Jumbo frames end-to-end? Or not?

Add me to the list of jumbo frames on iSCSI Initiators on 10Gbit.  Our heaviest workloads are SQL 2012 sequential reads.  At times our data warehouse VM will get close to 2 GB/sec if the cache hit ratio is near 100%.

This is probably the only scenario we have that gets the more mileage out of the additional payload size.

I would agree with Nick Dyer.  Test 1500 first, then move to 9000 if you really feel the calling to do so.

To whomever is interested:

One command I like to run on the guest VM to make sure the jumbo frame settings are correct in all areas is:  ping <array iSCSI IP> -f -l 8972                Don't use the iSCSI discovery IP. Use one of the actual data IPs.  And use a lowercase "L" for the second switch.

davecramp16
Occasional Advisor

Re: Jumbo frames end-to-end? Or not?

I find it's best to explain Jumbo Frames as a way to reduce workload of switches and network adapters rather than something that's really going to increase performance.

Increasing the packet size can reduce the number of packets required to send traffic, which reduces the workload of switches and other network equipment (adapters etc..) especially in a TCP world where the equipment runs checksums and other sequence checking on every packet.

This is a lot more visible on 10Gb networks where the number of packets per second will typically be much higher than that of a 1Gbit network.