Workstations dropping packets when pinged with MTU over 1000

[Craig A. Sharp]

We have several HP workstations that all seem to drop packets when they are pinged with an MTU over 1000.
This seems to be causing an issue with large ftp transfers to and from these boxes.

Any idea what is happening and how to correct it?

Thanks,

Craig

[Ron Kinner]

What NIC do you have? We had one type in several servers which was EMI sensitive and would randomly drop packets and mess up FTP transfers. Usually at the larger packet sizes.

Are the workstations all on the same subnet are is there a router in the way?

Ron

[Bill Hassell]

Check the LAN card settings with lanadmin -x as in lanadmin -x 0 (where 0 is the LAN ID of a specific LAN card from lanscan). Is the card 100Mbit and showing Auto (auto-negotiation) and also showing half-duplex? Most likely, the LAN card, switch and cable lengths have combined to form a timing error for auto-negotiation. You can verify this with lanadmin by look at the statistics for the card. FCS errors, framing errors and collisions all indicate failed negotiation.

Solutions:
make the cables much longer, or much shorter and see if negotiation is more reliable, or

lock the LAN cards into manual 100Mbits at full duplex and then change the switch to manual 100Mbuts to match.

[rick jones]

frist a terminology nit - the ping (aka ICMP echo) requests/responses have packet sizes, not MTU sizes. MTU's are attributes of interfaces, not packets.

the stuff about checking for FCS and late collisions and other errors in lanadmin is goodness.

however, if the machine were otherwise idle on the net, i wouldn't think a duplex mis-match would appear in a simple ping test - there wouldn't be a situation where both sides of the wire would be trying to transmit at the same time. of course if the system isn't otherwise idle on the net the ping test may indeed experience the duplex mis-match problems.

one very common cause of duplex mismatch is when someone say decides to hard-code some or all the switch ports, but does not hard-code all the NICs. when one side is hardcoded, autoneg will fail and when autoneg fails, the side trying to autoneg is required by the IEEE spec to go into half-duplex.

so, if you go the hardwire route, make sure you hardiwre EVERYTHING to the SAME value EVERYWHERE. otherwise, when someone helpfully tries to straighten-up the cables and moves them around on the switch, all hell could break loose.

i myself have led a charmed autoneg life - it has it seems always worked for me, but i may never have had the NIC and calbe lengths together at the same time...

[Craig A. Sharp]

Hi folks,

Sorry I took so long to get back. Got sidetracked :-).

I have checked the duplex on both the switch and workstations. Both match.

In using the command (pinging from box to switch):

ping 172.x.x.x 1500 -n 100

I am seeing about a 20-30% packet loss. If I ping from the switch to the box I get the same.

The NIC in the box is set to 1500 MTU.

If I change the command to:

ping 172.x.x.x 1000 -n 100

I have 0% packet loss. I can do this up to 1400. At 1500 I see the loss.

Additionally, if I try to use 1500 to ping a second time, I get 100% loss. I have to reset the connection and I am back to 20-30% loss.

This is happening on all our HP workstations but the SUN boxes are fine.

FYI: I am running HP-UX 11.00.

Thanks,

Craig

[Ron Kinner]

Your statement:

"If I change the command to:

ping 172.x.x.x 1000 -n 100

I have 0% packet loss. I can do this up to 1400. At 1500 I see the loss. "

rang a small bell. Is your switch doing any kind of tagging on vlans? Tagging adds extra bites which sometimes are enough to exceed the limit tho I would think it would be every packet and not just 30%. Cisco calls these "baby giant frames" and says they are often reported as errors. 802.1Q tagging adds 4 bytes to the frame, ISL (Cisco's) adds 30 bytes. Be interested to see how close to 1500 you can get. (Does it stop working at 1470 or 1496?)Normally you won't see the tags on a regular switch port as the switch strips them off. But if the port is set as a trunk or you have two switches trunked together and one doesn't know about the tagging you might see the tags at the port. I know some versions of hpux support tagging too. Perhaps you have turned it on by mistake. (Think it needs Auto-Port Aggegation to do tagging).

Can you take two of your HP's off line and connect them with a crossover cable and see if you have the same problem without the switch in the circuit? That would at least limit our trouble to a particular area.


Ron

[rick jones]

That 1400 is ok and 1500 not implies that there may be something related to fragmentation.

An ICMP header is (iirc) 8 bytes, and an IP header is 20 bytes, so all else being equal, I would guess that you would be OK up to a ping data size of 1500-28 or 1472 bytes.

So, you might reset the NICs, try 1472, see that it works, then try 1473 and see that it does not work.

You should also be looking to corellate the lost pings with drop stats in IP (netstat -p ip) and link-level errors with lanadmin.

[Craig A. Sharp]

On further investigation it seems that the problem is occuring at anything over 1000.

We are doing tagging but with the dropped packet at anything over 1000, this may not be the issue. Great suggestion though.

We are going to try hooking two workstations together with a crossover and see what happens.

[Ron Kinner]

Rick's comment about fragmentation sent me to ndd where there are a few possibilities:


ip_fragment_timeout:

Set the amount of time IP fragments are held while waiting for
missing fragments.

RFC1122 specifies 60 seconds as the time-out period for reassembly
of IP datagrams. This is a long time, but may be appropriate for
reassembly of datagrams that have traversed an internet. On local
file server systems, on the other hand, fragmentation reassembly
will either take place very quickly, or not all all, i.e., if all
fragments are not received at about the same time, it is likely
that one was dropped by the local interface, and will never
arrive. In this case, keeping fragments around for 60 seconds
may only exacerbate the problem. The actual value used is rounded
to the nearest second. [100, - ]
Default: 60000 (60 seconds)

ip_pmtu_strategy:

Set the Path MTU Discovery strategy: 0 disables Path MTU
Discovery; 1 enables Strategy 1; 2 enables Strategy 2.

Because of problems encountered with some firewalls, hosts,
and low-end routers, IP provides for selection of either
of two discovery strategies, or for completely disabling the
algorithm. The tunable parameter ip_pmtu_strategy controls
the selection.

Strategy 1: All outbound datagrams have the "Don't Fragment"
bit set. This should result in notification from any intervening
gateway that needs to forward a datagram down a path that would
require additional fragmentation. When the ICMP "Fragmentation
Needed" message is received, IP updates its MTU for the remote
host. If the responding gateway implements the recommendations
for gateways in RFC 1191, then the next hop MTU will be included
in the "Fragmentation Needed" message, and IP will use it.
If the gateway does not provide next hop information, then IP
will reduce the MTU to the next lower value taken from a table
of "popular" media MTUs.

Strategy 2: When a new routing table entry is created for a
destination on a locally connected subnet, the "Don't Fragment"
bit is never turned on. When a new routing table entry for a
non-local destination is created, the "Don't Fragment" bit is
not immediately turned on. Instead,

o An ICMP "Echo Request" of full MTU size is generated and
sent out with the "Don't Fragment" bit on.

o The datagram that initiated creation of the routing table
entry is sent out immediately, without the "Don't Fragment"
bit. Traffic is not held up waiting for a response to the
"Echo Request".

o If no response to the "Echo Request" is received, the
"Don't Fragment" bit is never turned on for that route;
IP won't time-out or retry the ping. If an ICMP "Fragmentation
Needed" message is received in response to the "Echo Request",
the Path MTU is reduced accordingly, and a new "Echo Request"
is sent out using the updated Path MTU. This step repeats as
needed.

o If a response to the "Echo Request" is received, the
"Don't Fragment" bit is turned on for all further packets
for the destination, and Path MTU discovery proceeds as for
Strategy 1.

Assuming that all routers properly implement Path MTU Discovery,
Strategy 1 is generally better - there is no extra overhead for the
ICMP "Echo Request" and response. Strategy 2 is available
only because some routers, or firewalls, or end hosts have been
observed simply to drop packets that have the DF bit on without
issuing the "Fragmentation Needed" message. Strategy 2 is more
conservative in that IP will never fail to communicate when using
it. [0,2] Default: Strategy 2


Ron