Network communication problem when packets out of order ......

[Gerard Moroney]

This is a really horrible problem and I hope someone out there can help !

This is the scenario:

Two unix servers with one running a security application (master) and the other server using this application (client).

The client is a HP 9000 series L-class server running HP-UX 11.00. The master is an RS6000 box running AIX 4.3.3.

I'm not allowed to talk much about the application :-( but can say that the master records access to certain accounts on the client.

Both servers run a daemon to establish communication. When a certain feature of the application is used a number of processes are started on the master and client using up ports on each. Under certain circumstances (as yet unknown) when the application is terminated on the client the FIN (?) signal does not reach the master. In other cases the application does not terminate properly in the telnet session and it has to be restarted.

What makes this problem particularly awkward is that the client and master have to go through a firewall and there are a limited number of ports available for communications. As a result, after a number of sessions fail to terminate successfully it is no longer possible for subsequent sessions to be open (no available ports) and the whole thing grinds to a halt.

So the problem is either network or firewall related.

Any ideas ? I've got someone monitoring the network and hope to be able to analyse the results over the next few days.

I've had our network people look into the problem and they have indicated that at the time when I experience the problem the packets are arriving at the server out of order. Is there any way to improve resilence on the client to this network behaviour ?

I'll add postings to this message as and when I find out more. I hope that what I find out as a result of this little adventure into the realm of networks will prove useful to others in the future ............

[Donald Kok]

Is it possible to put a client on the same subnet as the server, and (X, ssh)-connect to the client behind the firewall. This way you seperate the two possible problems.

You could even put the client-software on the server. It will use the sockets as if it was another machine.

When you got this working w/o problems it is time to find out which ports are used, and to try to make it work trough a firewall.

[Paul R. Dittrich]

The "packets...arriving...out of order" being part of the problem sounds very strange indeed if your application is TCP-based.

If the firewall does proxying I'd look at that. If possible, put a packet sniffer on both sides of the firewall and try to capture a single session going wrong.

HTH,
Paul

[Ron Kinner]

Which machine is leaving the connections up? Does the client try to kill a session and it remains up or does the master do the killing?

For the HPUX side of things see:

http://tcp-impl.grc.nasa.gov/tcp-impl/list/archive/2197.html

This will help get rid of sessions when the FIN gets lost. The default on an 11.0 is to leave them up forever.

There is also an NDD parameter you can tweak to decrease the amount of time it waits before resending an unack'd packet but I'd try the FIN_WAIT_II fix first.

Don't know much about AIX but I think the FIN_WAIT_II timer is about 11 minutes.

Ron

[rick jones]

Just for grins, make sure the L is up on the latest HP-UX 11 transport patches.

As for the fin_wait_timeout kludge added in 11, it should only be necessary if the apps do not actuall call close (and only do shutdown()) or if say some busted firewall software is responding to the keepalive probes when the remote end of the connection is actuall gone.

When an app calls close(), the tcp connection should enter a detached state, and the tcp_keepalive_detached_interval timer should kick-in to do keepalives whether nor not the app set SO_KEEPALIVE.

If the app uses abortive close, and the firewall looses packets, the RST could be lost. Some judisious use of tcpdump on the L and the AIX box might be in order.

[Joseph T. Wyckoff]

Not talking about HPUX here... I am an NT guy and don't really know.

In theory, if a TCP packet arrives out of sequence there are a few things that can happen...

Lets say packet number 6 arrives, and packet number 4 was expected

1. Packet 6 is dropped as gibberish

1a. a retransmit request is sent saying 'packet 3 was gotten successfully (ie please resend 4) but packet 6 is still dropped

2. Packet 6 is held 'a while' in hopes 4 and 5 will arrive

2a. a retransmit request is sent saying 'packet 3 was gotten successfully (ie please resend 4) - packet 6 is held a while

By the way, whether the packet is queued, or dropped immediately may not matter much, if the packet is held 'for a little while' say a few seconds, it will either time out and be dropped, or the buffers will fill, and packets get dropped.

---
The real question tho, is why is packet 6 arriving before packet 4.

This could be a routing issue (different routes with different latencies) or an issue in the firewall (different rules for different packets some may process quicker...)

Or all sorts of other issues, in principle (QOS, for example.)

Presumably there is a wide area network involved - so you might consider tuning things like your TCP window, retransmission retries - that sort of thing.

You might also consider pacing your application. You may benefit by slowing down its transmissions, or allowing fewer simultaneous connections if this is possible.

[harry d brown jr]

(1) Why is your client bigger and faster than your master ??

(2) Why aren't you running the newest AIX ??

(3) When was the last time you patched your HP system ??


live free or die
harry

[Eric Wilford]

As noted by others, check patches on your HP-UX 11.0 system. In particular there is a known problem when TCP on the HP-UX systems receives and "out of order" FIN that can leave the connection open. This is fixed in recent "cumulative ARPA Transport Patch".

Service request # 806132568

The HP Response Center can help locate the correct patch as well as review any network trace you may have.