topic Re: ftp FIN_WAIT_2 and CLOSE_WAIT in Operating System - HP-UX

ftp FIN_WAIT_2 and CLOSE_WAIT

pacific_1 — Mon, 17 Mar 2008 08:26:42 GMT

Hi everyone!

I have 2 servers in my office with services running on:
Server1 (rp3440): HP-UX 11.11 (FTP server / 10.8.7.22)
Server2 (rx2620): HP-UX 11.23 (FTP client / 10.8.7.32)

There are many many transfering file jobs running on these servers everyday. And last saturday, a guy in application team tell me that he find a problem on these servers:
And this is the results when I check:
On Server1 (FTP server):
# netstat -a | grep ftp
tcp 0 0 Server1.ftp 10.8.7.32.62505 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.62198 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.53139 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.62768 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.54354 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.62591 FIN_WAIT_2
tcp 0 0 Server1.ftp 10.8.7.32.53241 FIN_WAIT_2
And on Server2 (FTP Client):
# netstat -a | grep ftp
tcp 0 0 Server2.53241 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.53139 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.62768 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.62505 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.62198 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.62591 10.8.7.22.ftp CLOSE_WAIT
tcp 0 0 Server2.54354 10.8.7.22.ftp CLOSE_WAIT

Anyone has experienced with this problem? I don't known why? Please help me to fix it?

Thanks!

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

Laurent Menase — Mon, 17 Mar 2008 15:55:17 GMT

It justs means that your ftp client are not closing the socket, may be they are doing something else.

ftp uses 2 ports ftp and ftp-data, here it is ftp, so it is the command port which stays opened.

ftpd had proably timedout, and ftp is on something else.

if ftp is not working properly hang somewhere, disk access, dns access for instance,

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

Tim Nelson — Mon, 17 Mar 2008 16:40:20 GMT

In my past experiences at least with telnet I have found that HPUX and others like to send/see two closes ( i.e. fin_wait_1 and fin_wait_2 ). Mostly with windows products I have found that they exit on the first one and HPUX will wait for a about 2 hours before closing the second one.

This also happens with finely programmed applications ( typically windows ) that do not follow the proper close protocols.

You can adust the fin_wait timeout with ndd (man ndd) if it bothers you. These do not hurt anything.

close_waits are the same, the client did not close the connection properly, just disappeared and HPUX is still waiting for the final close.

you can test this. start up an ftp from A to B , identify it in netstat, then kill -9 the client on A(circumventing the proper close sequence).

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

skt_skt — Tue, 18 Mar 2008 00:55:28 GMT

MORE INFORMATION
TCP Connection States
Following is a brief explanation of this handshake. In this context the "client" is the peer requesting a connection and the "server" is the peer accepting a connection. Note that this notation does not reflect Client/Server relationships as an architectural principal.

1. Connection Establishment

â ¢ The client sends a SYN message which contains the server's port and the client's Initial Sequence Number (ISN) to the server (active open).
â ¢ The server sends back its own SYN and ACK (which consists of the client's ISN + 1).
â ¢ The Client sends an ACK (which consists of the server's ISN + 1).

2. Connection Tear-down (modified three way handshake).

â ¢ The client sends a FIN (active close). This is a now a half-closed connection. The client no longer sends data, but is still able to receive data from the server. Upon receiving this FIN, the server enters a passive close state.
â ¢ The server sends an ACK (which is the clients FIN sequence + 1)
â ¢ The server sends its own FIN.
â ¢ The client sends an ACK (which is server's FIN sequence + 1). Upon receiving this ACK, the server closes the connection.

A half-closed connection can be used to terminate sending data while sill receiving data. Socket applications can call shutdown with the second argument set to 1 to enter this state.
Back to the top

Netstat Output
The above TCP connection states can be monitored in a network trace under the TCP flags. It is also possible to determine the status of the connection by running the Netstat utility and looking at the State column. Netstat is shipped with Windows NT, Windows 95, and TCP/IP-32 for Windows for Workgroups.

State explanations as shown in Netstat:
State Explanation
------------ --------------------------------------------------------

SYN_SEND Indicates active open.

SYN_RECEIVED Server just received SYN from the client.

ESTABLISHED Client received server's SYN and session is established.

LISTEN Server is ready to accept connection.

NOTE: See documentation for listen() socket call. TCP sockets in listening state are not shown - this is a limitation of NETSTAT. For additional information, please see the following article in the Microsoft Knowledge Base:
134404 (http://support.microsoft.com/kb/134404/EN-US/) NETSTAT.EXE Does Not Show TCP Listen Sockets
FIN_WAIT_1 Indicates active close.

TIMED_WAIT Client enters this state after active close.

CLOSE_WAIT Indicates passive close. Server just received first FIN from a client.

FIN_WAIT_2 Client just received acknowledgment of its first FIN from the server.

LAST_ACK Server is in this state when it sends its own FIN.

CLOSED Server received ACK from client and connection is closed.
As an example, consider the following scenario:

A socket application has been terminated, but Netstat reports the socket in a CLOSE_WAIT state. This could indicate that the client properly closed the connection (FIN has been sent), but the server still has its socket open. This could be the result of one instance (among all threads or processes) of the socket not being closed.

NOTE: It is normal to have a socket in the TIME_WAIT state for a long period of time. The time is specified in RFC793 as twice the Maximum Segment Lifetime (MSL). MSL is specified to be 2 minutes. So, a socket could be in a TIME_WAIT state for as long as 4 minutes. Some systems implement different values (less than 2 minutes) for the MSL.

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

whiteknight — Tue, 18 Mar 2008 02:29:25 GMT

Pacific,

You got this error often ?

you can refer to this doc on how to clear the connection.

http://www12.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01003418-1

WK

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

Svetoslav Gyurov — Tue, 18 Mar 2008 07:50:31 GMT

Hi Pacific,

Probably year ago I also faced this problem and it was even worse, because I had hundreds of FIN_WAIT_2 connections and the server was unable to open and serve new connections. I found document in the KB of HP which resolved my problem.

This is the document:
How do I clear or prevent too many fin_wait_2 connections?
DocId: KBRC00015664 Updated: 5/30/06 12:57:00 PM
PROBLEM

Too many idle fin_wait_2 connections. How can I get them cleared out or to time out?

CONFIGURATION
hp9000 hp-ux 11.X
RESOLUTION

There is an ndd parameter, post-patch PHNE_19375/11.0 and included in 11i,
that is called

tcp_fin_wait_2_timeout

This parameter sets the fin_wait_2 timer on 11.x to stop idle fin_wait_2
connections. It will not survive a reboot, so modification of the
/etc/rc.config.d/nddconf is a necessary.

tcp_fin_wait_2_timeout specifies an interval, in milliseconds, after which the
TCP connection will be unconditionally killed. An appropriate reset segment
will be sent when the connection is killed.

The default for tcp_fin_wait_2_timeout is 0, which allows the connection
to live forever, as long as the far side continues to answer keepalives.

To enable the tcp_fin_wait_2 timer to timeout do the following:

1. To get the current value (0 is turned off):
# ndd -get /dev/tcp tcp_fin_wait_2_timeout
0

2. To set the value to 10 minutes:
# ndd -set /dev/tcp tcp_fin_wait_2_timeout 600000

3. Check the setting:
# ndd -get /dev/tcp tcp_fin_wait_2_timeout
600000

Note: (1000 ms in 1 second) * (60 seconds) * (10 minutes)= 600000 ms.
10 minutes is just an example but probably a good selection. Using a
setting less than 10 minutes is not recommended by HP and may cause data loss
with half-closed TCP connections.

Using the ndd on command line will not survive a reboot, so you need to update

/etc/rc.config.d/nddconf

with the parameter so that it will be set at boot time.

TRANSPORT_NAME[0]=tcp
NDD_NAME[0]=tcp_fin_wait_2_timeout
NDD_VALUE[0]=600000

Use this commend to read the nddconf file and implement the ndd's:

# ndd -c

NOTE: patches may be superceded, verify your patch levels via the patch
database on the IT resource Center.

Regards,
sve

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

skt_skt — Tue, 18 Mar 2008 09:44:08 GMT

yes, i did the same ndd setting too.

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

rick jones — Tue, 18 Mar 2008 16:25:20 GMT

Tim -

Those "two closes" are a standard, correct :) TCP connection tear-down sequence. Sadly, with its heritage of supporting limited numbers of TCP endpoints, Windows has "trained" applications to use an "abortive" close of connections. This can often cause problems and bypasses one of TCP's data integrity mechanisms.

Re: ftp FIN_WAIT_2 and CLOSE_WAIT

Laurent Menase — Thu, 20 Mar 2008 10:22:13 GMT

once again, if a side of the connection is in fin wait 2 and the other in close wait state (client side) we are still half closed, and the problem if there is one is that client side doesn"t close the socket for whatever question.

The real thing to identify there is why the applicatios doesn"t close the socket.

in case of rcp for instance, it is normal because the application is doing a half close.

335394 connections closed (including 48427 drops)

AndyHuang — Thu, 24 Dec 2015 07:01:22 GMT

Hi all, In the hp-ux netstat -p tcp output I have a question about "335394 connections closed (including 48427 drops)".

I want to know what including drops represent?Does it means the number of reset connections?