FIN_WAIT_2 Connections

[Warren griggs]

Hi,

I have a server running HPUX 11.00 that has a Microslop server connecting to it.

We seem to have a lot(407) of FIN_WAIT_2 connections when I run a netstat -a.

I've read that this is known problem from Microsoft servers and also the default timeout is 10 minutes for FIN_WAIT_2's on Solaris, is HPUX the same? Is it possible for me to change the default timeout to something like 3 minutes, if so what kind of impacts will this have and how do I do it?

If there are any other recommended solutions (short of hitting the Microsoft box with a hammer!)

Thanks,
Colin.

[James R. Ferguson · Accepted solution]

Hi Colin:

# ndd -set /dev/tcp tcp_fin_wait_2_timeout 600000

...since the timer's value is in milliseconds. You will need to add this to '/etc/rc.config.d/ndd.conf' make this change "permanent".

Regards!

...JRF...

[Massimo Bianchi]

Hi,
for the HPUX site you can change all this values using
ndd

there are a lot of other thread with full specs about the parameter and their use.

ndd -h supported

or

ndd -h unsupported

to have the simple list...

HTH,
Massimo

[Stefan Farrelly]

If you use ndd to change a tcp value dont forget to put it in /etc/rc.config.d/nddconf or else you will lose it next time you reboot.

[Warren griggs]

Thanks Guys.

Can I do this while the server is being used or do I need to do this at a quiet time?

Colin.

[Stefan Farrelly]

You can use ndd on the fly - no need to do at a quiet time.

[S.K. Chan]

First you can find out what's the current value by running ..
# ndd -get /dev/tcp tcp_fin_wait_2_timeout
The /etc/rc.config.d/nddconf file would look like this ..(if you want to set to to say 30mins)..
TRANSPORT_NAME[0]=tcp
NDD_NAME[0]=tcp_fin_wait_2_timeout
NDD_VALUE[0]=1800000

[Cole L.]

I found (not author) this script useful. We use a program that will not restart if there are any fin_wait_2 states on a specific port.

...

Run the script, at the log produced contains the exact line to run to kill off an offending item. It's nice.

#
# Temporary files used to compare netstat output
#
MYSCRIPTNAME=${0##*/}
TMPFILE1=/var/tmp/$MYSCRIPTNAME.1
TMPFILE2=/var/tmp/$MYSCRIPTNAME.2

#
# Create a log file to keep track of connection that were removed
#
LOGFILE=/var/adm/$MYSCRIPTNAME.log


function getFinWait2 {

/usr/bin/printf "%.2x%.2x%.2x%.2x%.4x%.2x%.2x%.2x%.2x%.4x\n" $(/usr/bin/netstat -an -f inet | /usr/bin/grep FIN_WAIT_2 | /usr/bin/awk '{print $4,$5}' | /usr/bin/sed 's/\./ /g') > $TMPFILE1
}

function compareFinWait2 {

FIRST_TIME=1

cp $TMPFILE1 $TMPFILE2
getFinWait2

comm -12 $TMPFILE1 $TMPFILE2 | while read CONN
do
if [[ $CONN != "000000000000000000000000" ]]
then

if [ $FIRST_TIME -eq 1 ]
then
print >> $LOGFILE
date >> $LOGFILE
FIRST_TIME=0
fi

print "/usr/bin/ndd -set /dev/tcp tcp_discon_by_addr
\"$CONN\""
>> $LOGFILE
/usr/bin/ndd -set /dev/tcp tcp_discon_by_addr $CONN
fi
done

getFinWait2
}

#
# Main
#

touch $TMPFILE1
touch $TMPFILE2

compareFinWait2

[Warren griggs]

Hi,

I have ran

# ndd -get /dev/tcp tcp_fin_wait_2_timeout

and the output is 0

Does this mean I have no current setting or does it mean that it is actually 0.

If so is it worth me changing it to 3 minutes?

Also will this have an impact on anything else?

Thanks,
Colin.

[James R. Ferguson]

Hi COlin:

A value of zero (0) signifies an infintite wait. Values of around 10-minutes (or more) are recommended.

Regards!

...JRF...

[James A. Donovan]

0 means that the connecgtion will wait indefinitely to timeout. Since you are having problems, you want to set this value to something like 5 minutes (600000).

$ ndd -h tcp_fin_wait_2_timeout

tcp_fin_wait_2_timeout:

Determines how long a TCP connection will be in FIN_WAIT_2.

Normally one end of a connection initiates the close of its end
of the connection (indicates that it has no more data to send) by
sending a FIN. When the remote TCP acknowledges the FIN, TCP
goes to the FIN_WAIT_2 state and will remain in that state until
the remote TCP sends a FIN.

If the FIN_WAIT_2 timer is used, TCP will close the connection
when it has remained in the FIN_WAIT_2 state for the length of
the timer value.

The FIN_WAIT_2 timer must be used with caution because when TCP
is in the FIN_WAIT_2 state the remote is still allowed to send
data. In addition, if the remote TCP would terminate normally
(it is not hung nor terminating abnormally) and the connection is
closed because of the FIN_WAIT_2 timer, the connection may be
closed prematurely. Data may be lost if the remote sends a
window update or FIN after the local TCP has closed the
connection. In this situation, the local TCP will send a RESET.
According to the TCP protocol specification, the remote TCP
should flush its receive queue when it receives the RESET. This
may cause data to be lost.
[0-2147483647 Milliseconds] Default: 0 (indefinite)

[Warren griggs]

When I changed this value I saw a significant drop in the number of FIN_WAIT_2's. Would the OS known which ones were over 10 minutes old and drop these as soon as the value changed?

[James R. Ferguson]

Hi (again) Colin:

Yes, reducing the 'tcp_fin_wait_2_timeout' value would mean that connections exceeding this timeout would procedure to closure.

Regards!

...JRF...

[James A. Donovan]

yup.

[rick jones]

From the standpoint o fthe transport, 400 FIN_WAIT_2 connections is nothing. It could handle orders of magnitude more than that without breaking a sweat.

Now, if those FIN_WAIT_2's are associated with open file descriptors (that is the app has only called shutdown() and not close(), then it could be an issue for the app. However, if the app has called close(), it is only a non-issue for the transport.

If the app calls close(), then the tcp_keepalive_detached_interval will come into play - after a default of two minutes, keepalive probes will be sent for up to tcp_ip_abort_interval milliseconds. If no response to the keepalive probe, then the connection is terminated.

If there are responses to the probes, it implies that the other side did not do the typical, kludgey abortive close, but is simply not remembering to call shutdown or close - IE an application bug on the other side.

It should be _really_ rare that the tcp_fine_wait_timeout kludge (imo) should be required - that is only required to workaround application bugs.

Also, while I'm on the soapbox, one should _NEVER_ need to use the tcp_discon stuff. That is a massive kludge that should have never left the lab. All it takes is nuking the wrong connection and knuth only knows what might happen. All it does is treat symptom, not root cause, and in treating symptom, it delays resolution of root cause...

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.txt