Handling of mis-ordered packets

[chuckk281]

There was a lot of discussion around how different networking products handle out-of-order pacfkets. Feel free to join the discussion.

Steve had a customer issue:

We have a customer that has seen a network throughput hit caused by the Cisco firewalls not forwarding packets in the order they are received. There is a Cisco forum post that describes this behaviour and the fix - https://supportforums.cisco.com/message/3245176 

The strange thing is that while this caused the BL465G6 to suffer a hit, BL460G7 servers running W2K8 R2 were able to cope with these mis-ordered packets without degredation. 

Can anyone point to a specific 'fix' in either the NIC or driver that would allow the G7 to cope much better than a 465G6?

Rick replied:

Both running the same rev of Windows?

Just how big a "hit" are we talking about here?  If there isn't actual packet loss, and just reordering, the first "hit" would be the added CPU overhead of the immediate ACKnowledgements on each out-of-order segment received.

If there were four out of order segments, the duplicate immediate ACKs that the receiver would send could trigger a spurrious fast retransmit at the sender, who will also adjust his cwnd - that could be a non-trivial throughput hit.

Speculating...

I suppose if the NIC and driver in the G7 have large receive offload enabled, and the G6's NIC and driver does not, and the LRO functionality in the G7's LOM is sufficiently "smart" it may be fixing the ordering of the TCP segments before passing them up to the host.

A very simple LRO implementation would only see if the newly arriving segment was the next expected one, and if it wasn't send it and the queued stuff up the stack.  Heck, if it was simple/stupid enough, it might even further re-order traffic.  For example, assume the NIC was holding segment 2, and expecting segment 3.  If it then got segment 1, the NIC/driver might be stupid enough to send segment 2 up the stack first and then segment 1.

A more sophisticated LRO might be holding segment 2, see that segment

1 comes right before, and "concatenate" them - ie. it might be willing to concatenate in both "directions" in the TCP sequence space.

The way to check that would be to take a packet trace on the receiving systems, and the wire leading to them (ie two traces, one on the network just before the NIC, and one in the receiving host, and, see what the traffic looks like in each case.

There was another link off there that talks about how the Cisco device will randomize (shift) the TCP sequence numbers from one side of the device to the other, and how that breaks SACK (because they don't/didn't rewrite the SACK blocks like they do the rest of the header).  Upon reading that, I think my characterization of them as "naughty" was diplomatic.

And I'm still not quite sure what value that randomization brings in the first place - they cannot be completely randomizing the sequence numbers - all a fixed shift does is mean that someone trying to inject does so in the shifted sequence space...

**********************

Join the discussion.