weird FIN_WAIT_2

[xray1111]

Hi, everybody

I encountered a strange situation. We have a tuxedo server deployed on a HP server (11.11), and we have our client application on remote PC which runs WinXP. And the staff who use the client program on PC always complain that the application runs too slowly.

So I use tcpdump tool filter the TCP packet between the client and the server, and found that sometimes the initial connection made by client didn't complete (not always), in this situation, the client didn't send FIN to server but only reply ACK according to the FIN packet sent by server. And the server keep trying to send FIN to client every 2 minutes for ever.

I wander why this problem could be happened, we know that the 8889 port of the tuxedo server is only a listen port, this port only respond to simple connection request and not mean to transfer application data after all.

09:50:22.676933 IP 135.133.23.61.1437 > 135.129.22.189.8889: S 3246342540:3246342540(0) win 65535
09:50:22.676983 IP 135.129.22.189.8889 > 135.133.23.61.1437: S 2409454420:2409454420(0) ack 3246342541 win 32768
09:50:22.746800 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 1 win 65535
09:50:22.747440 IP 135.133.23.61.1437 > 135.129.22.189.8889: P 1:33(32) ack 1 win 65535
09:50:22.747885 IP 135.129.22.189.8889 > 135.133.23.61.1437: P 1:397(396) ack 33 win 32768
09:50:23.027960 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 397 win 65139
09:50:28.006996 IP 135.129.22.189.8889 > 135.133.23.61.1437: F 397:397(0) ack 33 win 32768
09:50:28.036575 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 398 win 65139
09:52:29.896762 IP 135.129.22.189.8889 > 135.133.23.61.1437: F 397:397(0) ack 33 win 32768
09:52:29.933538 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 398 win 65139
09:54:31.789142 IP 135.129.22.189.8889 > 135.133.23.61.1437: F 397:397(0) ack 33 win 32768
09:54:31.799323 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 398 win 65139
09:56:33.681579 IP 135.129.22.189.8889 > 135.133.23.61.1437: F 397:397(0) ack 33 win 32768
09:56:33.692432 IP 135.133.23.61.1437 > 135.129.22.189.8889: . ack 398 win 65139
......

[Wouter Jagers]

I remember a similar situation. In my case it was due to crappy windows clients which refuse to adhere to standards.. I used to schedule a script to finish them off, but that might not be the cleanest way to do this.

I've found this in http://docs.hp.com/en/B2355-90748/B2355-90748.pdf , maybe you could check that out:

TCP FIN WAIT 2 Timer
The ndd TCP parameter tcp_fin_wait_2 determines how long a TCP
connection will be in FIN_WAIT_2.
Normally one end of a connection initiates the close of its end of the
connection (indicates that it has no more data to send) by sending a FIN.
When the remote TCP acknowledges the FIN, TCP goes to the
FIN_WAIT_2 state and will remain in that state until the remote TCP
sends a FIN.
If the FIN_WAIT_2 timer is used, TCP will close the connection when it
has remained in the FIN_WAIT_2 state for the length of the timer value.
The FIN_WAIT_2 timer must be used with caution because when TCP is
in the FIN_WAIT_2 state the remote is still allowed to send data. In
addition, if the remote TCP would terminate normally (it is not hung nor
terminating abnormally) and the connection is closed because of the
FIN_WAIT_2 timer, the connection may be closed prematurely.
Data may be lost if the remote sends a window update or FIN after the
local TCP has closed the connection. In this situation, the local TCP will
send a RESET. According to the TCP protocol specification, the remote
TCP should flush its receive queue when it receives the RESET. This
may cause data to be lost.
Default: 0 (indefinite)
Range: 0 - 2147483647
Units: Milliseconds


Cheers
Wout

[rick jones]

For how long a "for ever" did you wait? Ostensibly, if the UX server doesn't like the ACK's it is getting from the client, its end of the TCP connection will be in FIN_WAIT_1, which is an active retransmission state that should last no longer than tcp_ip_abort_interval

IIRC, the fin_wait timer kludge shouldn't kick-in when the connection is in FIN_WAIT_1.

Is the packet trace being taken on the UX system or someplace else?

I'd first make sure the system was up on the latest transport patch(es), and if the problem persisted start checking things like netstat stats and netstat -an to see what state the connection is in.

The 64bit question of course is why the UX system either does not see, or does not like the ACK of its FIN.

[xray1111]

problem was solved.

A router between the two peer filters some packets from the server to the client, and this cause the abnormal behaviour of the client application, and it didn't send the FIN segment which it should prior to the server.

we didn't set the tcp_fin_wait_2 parameter, so the server will maitain half close status until the client application exit.

thanks a lot

[rick jones]

This is what happens when intermediate devices start messing with things which are supposed to be end-to-end...