topic Re: Setup jumbo frames in Operating System - HP-UX

Setup jumbo frames

Wim Rombauts — Thu, 16 Oct 2008 08:45:48 GMT

I would like to configure jumbo frames on some of my 1Gb network cards.
I know where to configure this (/etc/rc.config.d/...) and what (MTU=9000), but
since I am not a network guru, I still have some questions :

- The network interface I want to configure is mainly for one-to-one communication with another system (2-node Oracle RAC). Do I need to configure and enable jumbo frames on both systems at the same time ? Or can I first do system 1 and later system 2 while they negotiate the maximum possible MTU for their communication automatically ?
- How do I activate this new MTU on the card. Of course, I can reboot the system, but what are the other options ? Is it possible to change the MTU-size on-line ? Without disrupting communication ?

Re: Setup jumbo frames

Jannik — Thu, 16 Oct 2008 08:58:48 GMT

The online part:
lanadmin -M 9000 ppa#

This is a link to some possibilities:
http://docs.hp.com/en/AB545-90001/ch01s09.html

Re: Setup jumbo frames

Ganesan R — Thu, 16 Oct 2008 09:07:32 GMT

Hi,

You can change the MTU size online without any impact by using SAM as well as #lanadmin -M ppa command.

But the changes will last till you reboot the server. To make it permenant you should be added to nddconf file.

Re: Setup jumbo frames

Wim Rombauts — Thu, 16 Oct 2008 09:09:45 GMT

OK? the lanadmin -M was 1 part of the answer that I was looking for.

Thus, the remaining question : What happens if oone system has already MTU=9000 and the other one does not (yet) ?
This may be a split-second situation, but will it disrupt network communication ?

Re: Setup jumbo frames

Ganesan R — Thu, 16 Oct 2008 09:12:15 GMT

Hi,

Followup to my previous update. To make it permenant you should change it on corresponding interface driver file.

Re: Setup jumbo frames

Jannik — Thu, 16 Oct 2008 13:33:29 GMT

http://sd.wareonearth.com/~phil/jumbo.html

It is acceptable to have one end point using jumbo frame whilst the other does not.

Jumbo frame is a strange thing - so to speak a "non standard" standard - and that has led to differences in implementation - that can complicate things, but, essentially the two end systems should negotiate an acceptable frame size based on MSS - maximum segment size.

From this document:
http://forum1.netgear.com/showthread.php?t=27782

Re: Setup jumbo frames

Jannik — Thu, 16 Oct 2008 13:35:05 GMT

Is the switch configured to use MTU 9000 as well?

Re: Setup jumbo frames

Court Campbell — Thu, 16 Oct 2008 14:14:14 GMT

Everything in the chain will need to have jumbo frames setup for it to work.That means other servers, switches, etc. You'll want to do some testing to make sure that it is working after you set it up. Try pinging with a large packet size and see if it gets chuncked into smaller frames.

Re: Setup jumbo frames

rick jones — Thu, 16 Oct 2008 23:47:51 GMT

Broadly speaking, if JumboFrames are set on one system in a broadcast domain it "should" be set on all systems in a broadcast domain.

And indeed, all the switches in the broadcast domain really must have JumboFrames enabled.

If "all" the traffic is TCP then you can get lucky if you enable JumboFrames on only some systems thanks to TCP exchanging Maximum Segment Size information at the time a TCP connection is established. The smaller of the two MSS options is what will be used.

If, however, you have to JF-enabled systems joined by a non-JF enabled switch, the MSS negotiation will not help you as the switch is not involved in the MSS negotiation.

Similarly if you are using UDP and sending from the JF-enabled system to/through non-JF enabled kit, any datagrams larger than the smaller MTU will vanish on the network. The sending system will IP fragment based on its JF-enabled MTU, and that will result in Ethernet frames beyond the limits of the non-JF enabled kit and so will be dropped.

PathMTU discovery will not help within the layer-2 broadcast domain...

Re: Setup jumbo frames

Wim Rombauts — Fri, 17 Oct 2008 06:15:12 GMT

I thought I was there, ready to close this thread, but when I attempt to change the MTU size with "lanadmin -M 9000 3", I get the following message :

Unable to change MTU size: errno = 16

I have been searching a little, but cannot find any explanation for this error number.

Re: Setup jumbo frames

Jannik — Fri, 17 Oct 2008 09:24:49 GMT

I have looked around andd found this document:
http://docs.hp.com/en/J1644-90018/ch04s02.html

It is an old document but a could not find another explanation.

it says that the error code 16 is reported if the wrong ppa is chosen... you use the ppa 3 could you sent lanscan and netstat -in.

It might be that the card is in a wrong state.

error:
If the error code is Errno=(6,22,16), the NMID used by lanadmin does not correspond to an active LAN card

Re: Setup jumbo frames

Wim Rombauts — Fri, 17 Oct 2008 10:19:54 GMT

Well, since this lan card is part of a single-lan APA aggregate (a preparation for a later extension), that could be an explanation for the errno = 16
When I try to set MTU=9000 on the APA Aggregate, I get errno = 223.

I will probably have to remove the lan card from the aggregate (lanadmin -X -d ?) but hen I have an empty aggregate ... ?
And can I just plain add the card back to the aggregate with lanadmin -X -a ?

I am rather unfamiliar with using lanadmin for APA configuration.

Re: Setup jumbo frames

Jannik — Fri, 17 Oct 2008 10:51:08 GMT

By default, TSO is disabled. To enable TSO on each specific interface, see the Ethernet Support
Guide, available in http://www.docs.hp.com, in the Networking and Communication section.
To verify if TSO is supported on an link aggregate or failover group, enter the following command:
# nwmgr -A vmtu -I instance -S apa
lan900 current values:
Virtual Maximum Transmission Unit = 0
In this example, a zero value indicates TSO is not enabled.

From this document page 16:
http://docs.hp.com/en/J4240-90045/J4240-90045.pdf

Re: Setup jumbo frames

Wim Rombauts — Fri, 17 Oct 2008 11:22:33 GMT

The correct sequence of commands is :
lanadmin -X -d 3 903
lanadmin -M 9000 3
lanadmin -X -a 3 903

I did this while the complete Oracle RDBMS software stack was down on one system to prevent communication during the change.
After the change, I started the clusterware software and that went fine.

Unfortunately, when I started the ASM instance, the ASM instance on the other system went down, taking the database down with it.

Fortunately, I was doing this on a test system.

After the total go-down, I could start the ASM instance and the database instance without issue.

Re: Setup jumbo frames

rick jones — Fri, 17 Oct 2008 17:21:07 GMT

TSO requires that CKO be enabled. JF does not require that CKO be enabled. CKO is disabled by default in HP-UX for 1Gig interfaces. CKO is enabled by default in HP-UX for 10Gig interfaces...

TSO can be thought of as a "poor man's JumboFrames" as it will reduce the number of times one goes up and down the protocol stack on the sender. However, since the same (small) size frames end-up on the network, it does nothing for the reciever. Whether that matters is one of those find "it depends" sorts of things. Some stacks (eg Linux) and NICs (10G mostly) are starting to support LRO - Large Receive Offload. Not sure when/if that will appear in HP-UX. Submit those enhancement requests early and often on that one I suspect is the best course there.

TSO/LRO can "offload" upwards of 64K at a time and so in theory, in combination can do more than just JF. Putting JF and TSO/LSO together starts into the realm of diminishing returns.

Re: Setup jumbo frames

Wim Rombauts — Fri, 06 Mar 2009 07:57:10 GMT

I see this thread is still open. My questions have been answered and I assigned points to the last questions.

Thank you all.