CLOSE_WAIT

[Medavie]

I am on HPUX 11.11 and would like to know if it is possible to terminate tcp connections in a CLOSE_WAIT state.

My tcp_keepalive_detached_interval is 120000
My /dev/tcp tcp_keepalive_interval is 7200000

The connections have been around for over 5 hours.

Thanks Bill Bird

[Ron Kinner]

See if the following answers your question:

From:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?admit=716493758+1092683988610+28353475&threadId=647225


"Use EXTREME caution in using ndd to shutdown socket connections


QUESTION
A powerful function of ndd in 11.x is the ability to "disconnect" a TCP connection and/or "clean-up" a TCP socket. This is done with one of the following ndd "tunables":

tcp_discon - Terminate a TCP connection
tcp_discon_by_addr - Terminate a TCP connection
tcp_fin_wait_2_timeout - Close idle fin waits

Please use and recommend these with GREAT CAUTION!


ANSWER
Typically these get used when someone is wanting to know "how do I clean up a socket?". There is always more to the story that needs to be learned before using ndd.

The reason for great concern is twofold. The first is that we need to address the "real" issue of why these socket connections are "hanging around" and not just cover it up by severing it. The real issue in these cases is almost always an application issue or a operational issue. It is rarely a network issue. The second reason is that use of the tcp_discn* options MAY cause data loss or corruption!

Looking at the most common socket "states":

ESTABLISHED:

If a socket is in an ESTABLISHED state, we clearly should not be blowing it away as there is still an application running that has the connection open. If this is the case, work on the application issue, and if needed, kill the application, not the socket.

CLOSE_WAIT:

In this case there is still a local application that has the connection open and the same principal applies as does the "ESTABLISHED" case. Often the user will swear that the application has been terminated. I have yet to see a case where this is true. It is sometimes the case that the main program that created (or accept(2)ed) the socket has terminated but a child is still referencing the socket. In any case, the public domain lsof(1) command is your friend in determining which process(s) is still referencing the socket.

Once this is determined, the customer (or the application provider) will need to determine if the application is supposed to still have the socket open and, if not, what the application problem is. It may be the case that the application has yet to finish reading data from the socket. If you "kill" the socket, the data is "killed" as well. Not a good thing to do unless you know the application.

Do NOT rely on the output of netstat -an to see if there is unread data on the socket because 11.x will almost always report 0 bytes of data in the Recv-Q even when data is present.

FIN_WAIT* (FIN_WAIT_1/FIN_WAIT_2):

This state is actually the "other end" of a connection that is in CLOSE_WAIT (no they are not "the same"!) In the case of a FIN_WAIT* state, the local application has done a "final close(2)" on the local socket, that is, all processes that may have referenced the socket are now gone. In this case, the local TCP has send a TCP packet to the remote with the FIN flag send (often called a "FIN packet"). The remote may have acknowledged the FIN flag (FIN_WAIT_2) or not (FIN_WAIT_1). In either case, the other end has not yet indicated that it is "done" with the connection.

This is actually a very unique state(s), in that a socket may be in this state and will have no application referencing it. In this case, and this case alone, lsof will NOT show an application that has a file descriptor pointing to the socket. The local TCP is now just waiting for the other end of the connection to tell us it has read all the data and is done with the connection as well. It is never appropriate to shutdown such a connection without knowing what is going on with the other end of the connection. IF you know the situation at the other end of the connection (e.g. the remote system is shutdown), then you may use ndd if you have real need to.

There have been issues where the application on the other end of the connection took over 30 minutes to do what it needed to do and close the other end of the connection. In one particular case where this was true, the local socket (FIN_WAIT_2) was "shut down" after 10 minutes and the remote aborted. The job had been running for almost a week on 10 interconnected hp9000s; the whole job had to be restarted from scratch.

This brings up the point of, why do you "need" to "shutdown" the socket?

Usually this is for one of two reasons. A real problem is when trying to start or restart an application (typically a server process) we get an error from the bind(2) system call indicating that the address is "in use". This usually indicates that we are trying to bind a TCP port number to the socket that is already being used.

This may be a time when you will need to use ndd to "shutdown" a socket that is in FIN_WAIT*. HOWEVER... it also indicates that the server process is not really written in the manner it "should be". A server that binds a specific port number to a socket (most do) should also be using "setsockopt(s, SO_REUSEADDR)" to let the port number be "reused". This is the way most well written server apps work and then it does not matter if there is an "old" socket still around in a state such a FIN_WAIT*. The server will still be able to (re)start.

So please DO take careful note of what ndd (and the lab) says about the tcp_discon* variables:


>> UNSUPPORTED ndd tunable parameters on HP-UX:
>> -------------------------------------------
>> This set of parameters are not supported by HP and modification of
>> these tunable parameters are not suggested nor recommended. "

Also see Dave Olker's explanation on the same page where he explains exactly how to use ndd to kill a connection.

Ron