TCP Segment size

[RafS]

Hi all,
I've a machine:
Model: "ia64 hp server Integrity Virtual Machine"
Release: HP-UX B.11.31
Machine: ia64

and I've an issue with my lan.

#netstat -ivwn
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
...
lan2 9000 172.21.3.0 172.21.3.2 60752 0 61920 0 0
lan1 1500 192.168.172.0 192.168.172.120 14431 0 14335 0 0
lan0 1500 192.168.171.0 192.168.171.141 75 0 361 0 0

1) lan2 has MTU=9000
#traceroute -s 172.21.3.2 -r -F 172.21.3.1 9000
traceroute to 172.21.3.1 (172.21.3.1) from 172.21.3.2, 30 hops max, 9000 byte packets
1 * * *

2) lan1 has MTU=1500

#traceroute -s 192.168.172.120 -r -F 192.168.172.119 9000
traceroute to 192.168.172.119 (192.168.172.119) from 192.168.172.120, 30 hops max, 9000 byte packets
1 192.168.172.120 (192.168.172.120) 0.117 ms !F 0.055 ms !F 0.034 ms !F

3) lan0 has MTU=1500
#traceroute -s 192.168.171.141 -r -F 192.168.171.140 9000
traceroute to 192.168.171.140 (192.168.171.140) from 192.168.171.141, 30 hops max, 9000 byte packets
1 192.168.171.141 (192.168.171.141) 0.977 ms !F 0.057 ms !F 0.044 ms !F

Why lan2 behaves in a different way ?
How can I detect the problem ?
I don't know if the different behave depends on OS settings or on network/switches settings.

Thanks in advance.
Best Regards.
Raffaele

[Mel Burslan]

MTU size 9000 means you are using jumbo frames. Jumbo frames require cooperation from the network switch this port is connected to. If you are not the administrator of networking gear, you need to find the person in charge of those and and let them know that this network interface will use Jumbo frames and they need to modify the switch port it os connected to, accordingly. Not a Cisco expert myself, hence can not provide you what needs to be changed but something definitely needs to know about jumbo frame usage, otherwise it won't work.

[RafS]

Hi Mel,
thanks for your reply.

I had the same problem too when lan2 had MTU=1500. According to you, I think the problem depends on switch configuration, but I'm not sure.

Thanks again.
Regards

[Mel Burslan]

Raf,
if regardless of the frame size, you always had a problem with this interface, it is always a good idea to check the other end of the network cable, i.e., the switch port. It might be dead, it might be assigned to a wrong VLAN, it might be configured to use trunking, which I am assuming you are not using. One of many reason may cause this disconnect and if you are controlling only one side, the server side, of the problem, you can not make decisive conclusions where the problem resides.

[RafS]

The problem on this interface is that I have to transmit a packet size less than MTU.
i.e.
If I set MTU=1500
I can transmit a packet size less than 1497
(traceroute -s source -r -F target 1496 )
If I set MTU=9000
I can transmit a packet size less than 8997
#traceroute -s 172.21.3.2 -r -F 172.21.3.1 9000
traceroute to 172.21.3.1 (172.21.3.1) from 172.21.3.2, 30 hops max, 9000 byte packets
1 * * *
#traceroute -s 172.21.3.2 -r -F 172.21.3.1 8996
traceroute to 172.21.3.1 (172.21.3.1) from 172.21.3.2, 30 hops max, 8996 byte packets
1 * cosbedb1-priv (172.21.3.1) 0.726 ms 0.507 ms

while with other interface I can transmit a packet size greater than MTU. I would know if I can set OS parameters to get the same behaviour.

[Mel Burslan]

I am flashing back to some dark days in my past with the information in the last post. Unfortunately, dissecting the network packet loads is a little mystery to me. I remember there are a few bytes of difference between the packet size and and what you can actually put in that packet, but I can not remember why.

Configuration parameters for this are ndd related and Rick Jones of these forums is the expert on ndd configuration. I see from his profile that he is still active and posting. So, when he sees this post, I am sure he can tell you how you can change the things (or why you should not).

[rick jones]

All it means is that the card and driver for lan2 supports JumboFrames and someone ran either lanadmin or nwmgr to set the MTU to the JumboFrames size.

As for why the 9000 byte traceroute worked, that isn't a "problem" it is just IP doing what it is supposed to do. When handed a datagram larger than the next-hop MTU, IP will fragment it into fragments that fit in the next-hop MTU and transmit them. When the arrive at the destination, the destinations IP stack will reassemble the fragments into complete datagrams and pass them up the stack.

Also, thusfar, none of this has been the TCP Segment size.

[RafS]

Hi Rick,
thanks for your reply and sorry for the wrong subject.

"When handed a datagram larger than the next-hop MTU, IP will fragment it into fragments that fit in the next-hop MTU and transmit them. When the arrive at the destination, the destinations IP stack will reassemble the fragments into complete datagrams and pass them up the stack."
Why it appens only in lan0 and lan1 ?
Why only in lan2 I can't transmit a datagram larger than MTU ?


Thanks again,
Regards.

Raffaele

[Viktor Balogh]

Raffaele,

does lan2 work with smaller datagram sizes like e.g. 1500? What if you set the MTU size to 1500 for lan2? To debug this, you should try the following cases for lan2:

- datagram size 1500, MTU 9000
- datagram size 9000, MTU 9000 -> checked, seems not to work
- datagram size 1500, MTU 1500
- datagram size 9000, MTU 1500

Like Mel wrote it, for jumbo frames to work you need to set it on both sides. Maybe it's a problem on the switch side.

[RafS]

Hi Victor,

does lan2 work with smaller datagram sizes like e.g. 1500?
Yes, it works.
What if you set the MTU size to 1500 for lan2?
I can transmit datagram less than 1497.

To debug this, you should try the following cases for lan2:

- datagram size 1500, MTU 9000 -> OK

- datagram size 9000, MTU 9000 -> KO !
- datagram size 1500, MTU 1500 -> KO !
- datagram size 9000, MTU 1500 -> KO !

Thanks a lot for your help.

Regards,
Raffaele

[rick jones]

"All" stations in the same broadcast domain must have the same MTU. If your lan2 interface is connected to a switch (or system) that does not also have a 9000 byte MTU, what will happen is:

1) IP on your system fragments based on the MTU of lan2.

2) the "lan2" interface sends a 9000+ byte frame to the "link peer" (fancy, pithy way to say "the station at the other end of the link")

3) If the link peer also uses a 9000 byte MTU, happiness and joy, but if the link-peer does not, it will not receive the frame down at "layer 2" of the stack (http://www.isc.org/files/imagecache/product_mid/9layer.thumb_.png).

So, it can look like a abyss.

Now, here is where TCP segment size *can* enter the discussion If all the communications are TCP, and your lan2 interface is the only one with the larger MTU, then the TCP MSS exchange will paper-over the misconfiguration - the remote system(s) will send an MSS option based on their (presumably) 1500 byte MTU. Your system will send one based on its 9000 byte MTU. The smaller of the two is what is used. That means that even though your lan2 was configured with an MTU of 9000 bytes, the MSS used by TCP is such that it never gets that large - and so the frames sent will still be received by the link peer and go on their merry way.

[rick jones]

Some other considerations:

*) various firewalls may not pass IP datagram fragments

*) depending on the implementation of traceroute, (check the documentation/manpage) saying 1500 may not be the total size of the datagram and it may still be fragmented. If "datagram" here is the UDP datagram then the 1500 bytes will (probably - at least based on my read of a *linux* traceroute manpage) include the UDP header, but not the IP header (20 bytes for IPv4) and so will cause the message sent by traceroute to be fragmented when the IP MTU (against which the IP header counts) is 1500 bytes.

[RafS]

Hi Rick,
I appreciate your help.

"So, it can look like a abyss."
Perfectly !

"Now, here is where TCP segment size *can* enter the discussion If all the communications are TCP
...
the frames sent will still be received by the link peer and go on their merry way."
It doesn't happen, may be because traceroute doesn't use TCP. I'm right ?

As you could understand, I'm not a network expert.
I just noticed a different bahaviour from lan0/1 and lan2, so I want to see if it is possible to identify/solve the problem from OS side or I have to involve network specialist.
As I wrote previously, I had the same problem when lan2 had MTU=1500 and datagram lenght = 1500 bytes.
Seems that, only for lan2, datagrams can't be fragmented.

*) various firewalls may not pass IP datagram fragments
There aren't firewall between source and target

Thanks a lot,
Raffaele

[rick jones · Accepted solution]

Traceroute does not normally use TCP.

To what is the lan2 interface connected? Not everything has a 1500 byte MTU even, so if the "link peer" or something else in that broadcast domain has a smaller MTU, you have the same issues as when one station has 9000 and another 1500 - the "formula" (as it were) is the same, only the constants are different.

[RafS]

Thanks a lot !
Raffaele