Why decrease jumbo frames my net throughput?

[Walter Kießl]

Hi there,

I have two rp4440 running hpux11.11 with 2 1000BSx cards. On one card I have enabled MTU 9000 the other card is set to MTU 1500. Both cards are configured into different subnets within differen VLANs. Jumboframes are enabled on the switch.

I do the following tests:
- netperf -fM -H -tTCP_STREAM -> which reports a throughput of more than 100 MB/sec.
- netperf -fM -H -tTCP_STREAM -> which reports a throughput of less than 52 MB/sec.

Tests with other tools (ftp, dd, ttcp) show the same direction: the NIC with jumboframes is slowlier than the one without.

Has anybody an idea why the jumbo frames decrease network throughput?
Are there any configs I have missed or I can do to improve the throghput (e. g. kernel param, bufcache ...)?

Thanks for your help.

[Ron Kinner]

Jumbo frames are not of much use unless the complete path is jumbo capable. If the frame can't stay jumbo from end to end then somewhere along the line it has to be broken up into a bunch of smaller frames.

There is an ndd parameter that may have an effect on performance.

ndd -h ip_pmtu_strategy

for details.

Ron

[Walter Kießl]

Thanks Ron.

ip_pmtu_strategy is set to the default (=1). Both rp4440 are configured the same. There is only one switch between them which has jumboframes enabled.
If the first netperftest mentioned above is started with option -v2 then it reports a MSS of 1460 - the second reports a MSS of 8960. Does this indicate that the packets are not broken up into smaller peaces?

I'd appreciate any help!!

Walter

[rick jones]

You should include the full netperf output. I'll guess that you are still using the default window size on HP-UX of 32768 bytes.

If there is packet loss on the network, the systems using jumboframe (I'm ass-u-me-ing the netperf tests you list are on two different sets of systems - one pair with JF, one pair without...) may have to use timeout-based retransmissions, and the ones using the standard frame size may not.

That is a function of the number of segments the connection can have outstanding at one time - for the "fast" retransmissions there need to be at least 6 segments at one time. 32768/8192 is not at least 6.

You can check the hypothesis by taking netperf -p tcp snapshots before and after each test, and running them through beforeafter (ftp://ftp.cup.hp.com/dist/networking/tools/) - see if there are retransmissions (near the top of the output) and/or retransmission timeouts (nearer to the bottom) ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt

If that does show timeouts for the JF test, try adding test specific -s and -S options to use larger socket buffer and thus window sizes. I'd try 128K myself. You can then consider editing /etc/rc.config.d/nddconf to change the TCP defaults (ftp://ftp.cup.hp.com/dist/networkin/briefs/annotated_ndd.txt)

Also, make sure you have the latest FTP patches so you can use > 56KB windows in FTP with the -B option.

[N.S. Sundar]

I suppose you have a NIC A at 1500-byte MTU connected to a NIC B at 9000-byte MTU, through a Jumbo-frames-capable switch. The problem with this is that B will send frames upto 9000 bytes, but A cannot receive frames bigger than 1500 bytes.

Just curious, you seem to be saying that the throughput is higher with A as the receiver
(first case above) and B as the sender. One would expect the reverse. Can you please check?

[rick jones]

If one does TCP between a system with a 9000 byte MTU and one with a 1500 byte MTU, the MSS exchange of TCP will cause each TCP to use 1460 byte MSSes. Where problems with mismatched MTU's will arrise is UDP or raw Ethernet communications.

There, things will not just be slow, they won't work at all.

[Walter Kießl]

Again thanks to all.

To clarify my config: I have 1 rp4440 with 2 1000BSx cards. One card, configured in one subnet and one vlan has MTU 1500. The other card, configured an another subnet with another vlan has MTU 9000. This rp4440 is connected to one switch. For the tests I have another rp4440 configured the same. The tests described above was run from one rp4440 to the ohter, testing first the mtu-1500-Lannic then the mtu-9000-Lannic.

Rick is right. It does not work at all if you use UDP with different frame sizes.

I forwarded Ricks hints regarding window and socket sizes to my HP support. They declined the need of changing the default values. But they can't help me with the performance isue, too ...

Unfortunately I cannot do testing any more because I had to set the servers into production. So I disabled jumbo frames on my 2 rp4440 and on the switch to avoid any performance issues.

What I learned and what my HP support recommends: Don't use Jumbo Frames with hpux1111.

[rick jones]

At the risk of contradicting the official support folks, one of whom I am not, and at the risk of not knowing something they know, I would have to say that the suggestion that one should not use JumboFrames with 11.11 is a bit harsh.

If you do get a chance to experiment further, instead of changing the defaults via ndd, simply add some -s and -S options to the netperf commands:

netperf -fM -H -t TCP_STREAM -- -s 128K -S 128K -m 32K

and see where that takes you. At the same time, be sure to track the netstat -p tcp statistics and perhaps the lanadmin -g mibstats statistics.

All assuming, of course (and probably already addressed) that the switch actually supports full 9000 byte JumboFrames. Some switches' "JumboFrame" support does not go that high.

BTW, I'm not sure if it was back-ported to 11.11 or not, but up on 11.23 there is also support for "TSO" or TCP Segmentation Offload" which is something of a "poor man's" JumboFrame. The NIC accepts for transmission TCP segments >> 1500 bytes (larger than 9000 bytes even) and will segment them to the standard sizes - thus eliminating the need for a JF-capable switch.

This has _most_ of the performance benefit of JF for the sender. However, it does nothing for the receiver who is still receiving a stream of 1500 byte frames.

[Walter Kießl]

Hi Rick,

I'm very upset about my HP support.
After dealing for more than 2 weeks with very poor LAN/NFS performance, doing a lot of measuring and tuning, they told me that there is a patch addressing my JumboFrame issue and a lot of other TCP/UDP issues: PHNE_29947 for the IETHER driver.

As I experienced a very bad NFS/LAN performance without JumboFrames, too, I will give it a new try at our next scheduled downtime in january 2005. This time using your hints!

Thanks a lot!
Walter