Setup jumbo frames

[Wim Rombauts]

I would like to configure jumbo frames on some of my 1Gb network cards.
I know where to configure this (/etc/rc.config.d/...) and what (MTU=9000), but
since I am not a network guru, I still have some questions :

- The network interface I want to configure is mainly for one-to-one communication with another system (2-node Oracle RAC). Do I need to configure and enable jumbo frames on both systems at the same time ? Or can I first do system 1 and later system 2 while they negotiate the maximum possible MTU for their communication automatically ?
- How do I activate this new MTU on the card. Of course, I can reboot the system, but what are the other options ? Is it possible to change the MTU-size on-line ? Without disrupting communication ?

[Jannik · Accepted solution]

The online part:
lanadmin -M 9000 ppa#

This is a link to some possibilities:
http://docs.hp.com/en/AB545-90001/ch01s09.html

[Ganesan R]

Hi,

You can change the MTU size online without any impact by using SAM as well as #lanadmin -M ppa command.

But the changes will last till you reboot the server. To make it permenant you should be added to nddconf file.

[Wim Rombauts]

OK? the lanadmin -M was 1 part of the answer that I was looking for.

Thus, the remaining question : What happens if oone system has already MTU=9000 and the other one does not (yet) ?
This may be a split-second situation, but will it disrupt network communication ?

[Ganesan R]

Hi,

Followup to my previous update. To make it permenant you should change it on corresponding interface driver file.

[Jannik]

http://sd.wareonearth.com/~phil/jumbo.html

It is acceptable to have one end point using jumbo frame whilst the other does not.

Jumbo frame is a strange thing - so to speak a "non standard" standard - and that has led to differences in implementation - that can complicate things, but, essentially the two end systems should negotiate an acceptable frame size based on MSS - maximum segment size.

From this document:
http://forum1.netgear.com/showthread.php?t=27782

[Jannik]

Is the switch configured to use MTU 9000 as well?

[Court Campbell]

Everything in the chain will need to have jumbo frames setup for it to work.That means other servers, switches, etc. You'll want to do some testing to make sure that it is working after you set it up. Try pinging with a large packet size and see if it gets chuncked into smaller frames.

[rick jones]

Broadly speaking, if JumboFrames are set on one system in a broadcast domain it "should" be set on all systems in a broadcast domain.

And indeed, all the switches in the broadcast domain really must have JumboFrames enabled.

If "all" the traffic is TCP then you can get lucky if you enable JumboFrames on only some systems thanks to TCP exchanging Maximum Segment Size information at the time a TCP connection is established. The smaller of the two MSS options is what will be used.

If, however, you have to JF-enabled systems joined by a non-JF enabled switch, the MSS negotiation will not help you as the switch is not involved in the MSS negotiation.

Similarly if you are using UDP and sending from the JF-enabled system to/through non-JF enabled kit, any datagrams larger than the smaller MTU will vanish on the network. The sending system will IP fragment based on its JF-enabled MTU, and that will result in Ethernet frames beyond the limits of the non-JF enabled kit and so will be dropped.

PathMTU discovery will not help within the layer-2 broadcast domain...

[Wim Rombauts]

I thought I was there, ready to close this thread, but when I attempt to change the MTU size with "lanadmin -M 9000 3", I get the following message :

Unable to change MTU size: errno = 16

I have been searching a little, but cannot find any explanation for this error number.

[Jannik]

I have looked around andd found this document:
http://docs.hp.com/en/J1644-90018/ch04s02.html

It is an old document but a could not find another explanation.

it says that the error code 16 is reported if the wrong ppa is chosen... you use the ppa 3 could you sent lanscan and netstat -in.

It might be that the card is in a wrong state.

error:
If the error code is Errno=(6,22,16), the NMID used by lanadmin does not correspond to an active LAN card

[Wim Rombauts]

Well, since this lan card is part of a single-lan APA aggregate (a preparation for a later extension), that could be an explanation for the errno = 16
When I try to set MTU=9000 on the APA Aggregate, I get errno = 223.

I will probably have to remove the lan card from the aggregate (lanadmin -X -d ?) but hen I have an empty aggregate ... ?
And can I just plain add the card back to the aggregate with lanadmin -X -a ?

I am rather unfamiliar with using lanadmin for APA configuration.

[Jannik]

By default, TSO is disabled. To enable TSO on each specific interface, see the Ethernet Support
Guide, available in http://www.docs.hp.com, in the Networking and Communication section.
To verify if TSO is supported on an link aggregate or failover group, enter the following command:
# nwmgr -A vmtu -I instance -S apa
lan900 current values:
Virtual Maximum Transmission Unit = 0
In this example, a zero value indicates TSO is not enabled.

From this document page 16:
http://docs.hp.com/en/J4240-90045/J4240-90045.pdf

[Wim Rombauts]

The correct sequence of commands is :
lanadmin -X -d 3 903
lanadmin -M 9000 3
lanadmin -X -a 3 903

I did this while the complete Oracle RDBMS software stack was down on one system to prevent communication during the change.
After the change, I started the clusterware software and that went fine.

Unfortunately, when I started the ASM instance, the ASM instance on the other system went down, taking the database down with it.

Fortunately, I was doing this on a test system.

After the total go-down, I could start the ASM instance and the database instance without issue.

[rick jones]

TSO requires that CKO be enabled. JF does not require that CKO be enabled. CKO is disabled by default in HP-UX for 1Gig interfaces. CKO is enabled by default in HP-UX for 10Gig interfaces...

TSO can be thought of as a "poor man's JumboFrames" as it will reduce the number of times one goes up and down the protocol stack on the sender. However, since the same (small) size frames end-up on the network, it does nothing for the reciever. Whether that matters is one of those find "it depends" sorts of things. Some stacks (eg Linux) and NICs (10G mostly) are starting to support LRO - Large Receive Offload. Not sure when/if that will appear in HP-UX. Submit those enhancement requests early and often on that one I suspect is the best course there.

TSO/LRO can "offload" upwards of 64K at a time and so in theory, in combination can do more than just JF. Putting JF and TSO/LSO together starts into the realm of diminishing returns.

[Wim Rombauts]

I see this thread is still open. My questions have been answered and I assigned points to the last questions.

Thank you all.