FTP transfer is hanging

[James A. Donovan]

The Oracle developer here has written an FTP script which is supposed to automagically transfer a file. It does log in to the remote server fine, but it doesn't actually transfer the file. It DOES create a 0 length file of the same name on the remote server. The killer thing is that I have no problems transferring files back and forth to this server manually.

The only thing I can see that is odd is that I'm seeing several tcp ports in FIN_WAIT_2 states where the other side of the connection was the remote server.

##################################################################
# PROGRAM: scdtcftp #
# PURPOSE: This program takes the file name , host server name #
# where the file has to be transfered , the user id #
# password of the user to be logged in the remote server#
# the source directory and the destination directory #
# and transferes the file. #
# NOTE: shift 8-Skips 8 parameters. The first 8 parameters were #
# reserved by Oracle when the program was registered as a #
# concurrent request. #
# cut -f 2 -d """ was used to removed the " from the #
# paramters. For example, the HOST NAME was like "10.152. #
# 2.136" before the " was removed. #
#################################################################
set $1
shift 8
HOST_NAME=`echo $1 |cut -f 2 -d """ |cut -c 1-`
LOGIN_NAME=`echo $2 |cut -f 2 -d """ |cut -c 1-`
PASSWD=`echo $3 |cut -f 2 -d """ |cut -c 1-`
S_DIR=`echo $4 |cut -f 2 -d """ |cut -c 1-`
D_DIR=`echo $5 |cut -f 2 -d """ |cut -c 1-`
C_FILE=`echo $6 |cut -f 2 -d """ |cut -c 1-`
cd $S_DIR
ftp -niv $HOST_NAME << EOF
user "$LOGIN_NAME" "$PASSWD"
cd $D_DIR
put $C_FILE
quit
EOF

[Mike McKinlay]

Assuming the S_DIR is the source directory, is this an explicit or relative path? Add a pwd after the CD S_DIR to check to make sure you're in the right spot.

Use 1>> and 2>> parms with the script command to output STDOUT and STDERR to another file in order to troubleshoot this a little more.

[Mike McKinlay]

One other question: is this on the same network subnet or does the FTP go through a firewall of some kind? It may be that you can establish the session but not the data portion of the FTP due to policies on the firewall.

[James A. Donovan]

Mike,

1) S_DIR is an explicit path.
2) There is a firewall between the hosts.
3) Setting the 1>> and 2>> shows me

Connected to 10.152.2.136.
220 sony-sun-ftp-p-1.digisle.net FTP server ready.
331 Password required for xxxxxxx.
230 User xxxxxxx logged in.
250 CWD command successful.
200 PORT command successful.
150 Opening ASCII mode data connection for sku_seed_20010112.txt.

in the standard out.

3) A new development is that I can no longer transfer via ftp from the local server to the remote server at all. Not even when I type everything in by hand. It hangs at the same point.

4) I CAN transfer files to the remote host from another local server that I have here which goes through the same firewall.

I'm thinking that my problem is really related to the following:

# netstat -a | grep 10.152
tcp 0 0 corvette.59164 10.152.2.136.ftp ESTABLISHED
tcp 0 0 corvette.59007 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58979 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58904 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58876 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58856 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58840 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58801 10.152.2.136.ftp FIN_WAIT_2
tcp 0 0 corvette.58678 10.152.2.136.ftp FIN_WAIT_2


It looks as if the dead connections are not getting closed completely.

[rick jones]

You might try the FTP transfer in "passive" mode by issuing an passive command before trying to do the transfer. That might make it through the firewall.

The FIN_WAIT_2 state is a state TCP enters when it has sent a FIN to the remote and had the FIN ACKnowledged. It is now waiting for a FIN from the remote to complete the connection shutdown, at which point it will transition to TIME_WAIT.

I have heard of firewalls doing ugly things and leaving connections in FIN_WAIT_2 or other stats on the one side. HP-UX 11 though does have a mechanism that will in theory cull these - after tcp_keepalive_detached_interval, the connection will start sending TCP keepalive probes. If those probes elicit no response, eventually the tcp_ip_abort_interval is reached and the connection is closed. If the probes elicit a RST the connectoin is closed. If the probes elicit simple ACKs (say from a buggy firewall) the connection will remain in FIN_WAIT_2.

If anything though, the FIN_WAIT_2 connections might be symptoms of the problem, no the cause.

[James A. Donovan]

fyi...the remote side shows

$ netstat -a | grep 162.49.86.42
sony-sun-ftp-p-1.ftp 162.49.86.42.58678 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58679 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.58801 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58812 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.58840 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58841 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.58856 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58857 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.58876 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58878 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.58904 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.55499 162.49.86.42.58912 57344 0 8760 0 ESTABLISHE
D
sony-sun-ftp-p-1.ftp 162.49.86.42.58979 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.58988 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.59007 32768 0 8760 0 CLOSE_WAIT
sony-sun-ftp-p-1.ftp-data 162.49.86.42.59010 57344 0 8760 0 ESTABLI
SHED
sony-sun-ftp-p-1.ftp 162.49.86.42.59164 32768 0 8760 0 ESTABLISHED
sony-sun-ftp-p-1.ftp-data 162.49.86.42.59169 57344 0 8760 0 ESTABLI
SHED

[James A. Donovan]

After using ndd to set the tcp_fin_wait_2_timeout to 60 seconds all my FIN_WAIT_2 states cleared up, and the remote server dropped those ESTABLISHED connections on the ftp-data port.

Then I tried successfully to transfer the /etc/hosts file to the remote server.

Everythings great, right? Wrong...:-(

Local Side:
# netstat -a | grep 10.152
tcp 0 2920 corvette.59982 10.152.2.136.ftp-data ESTABLISHED
tcp 0 0 corvette.59980 10.152.2.136.ftp ESTABLISHED
# more /tmp/std.out
Connected to 10.152.2.136.
220 sony-sun-ftp-p-1.digisle.net FTP server ready.
331 Password required for sidftp.
230 User sidftp logged in.
250 CWD command successful.
200 PORT command successful.
150 Opening ASCII mode data connection for sku_seed_20010112.txt.

Remote Side: (trying to transfer sku_seed_20010112.txt)
$ netstat -a | grep 162.49.86.42;ls -la
sony-sun-ftp-p-1.ftp 162.49.86.42.59980 32768 0 8760 0 ESTABLISHED
sony-sun-ftp-p-1.ftp-data 162.49.86.42.59982 57344 0 8760 0 ESTABLI
SHED
total 1594
drwxr-xr-x 2 sidftp sony 512 Jan 12 19:56 .
drwxr-xr-x 10 sidftp ftp 3584 Jan 12 18:23 ..
-rw-rw-r-- 1 sidftp sony 253977 Jan 4 20:01 sku_seed_20001109.txt
-rw-rw-r-- 1 sidftp sony 254547 Jan 4 01:13 sku_seed_20001127.txt
-rw-rw-r-- 1 sidftp sony 257617 Jan 5 19:07 sku_seed_20010105.txt
-rw-rw-r-- 1 sidftp sony 0 Jan 12 19:56 sku_seed_20010112.txt

Anyone have any ideas?

[Steven Sim Kok Leong]

Hi,

You do not seem to be using passive FTP. Without passive FTP, the firewall has to allow a range of ports at your FTP client to be accessed by your FTP server. This is because a reverse connection is established for FTP's data connection.

I believe that your firewall only allow a specific range of ports to be accessible at your FTP client. When you have few TCP connections at your FTP client, the port number (>1023) would be small. when you have many TCP connections at your FTP client, the port number would be large.

Whether your FTP transfer between the FTP client and FTP server is successful is very much dependent on whether this port number at your FTP client falls within the range allowed at your firewalls or not.

To workaround this problem, request your firewall administrator to open up a larger range of ports for FTP for ftp-data. There should be two rules. One existing for connection to port 21 on the FTP server. The other rule for return ftp-data connections to port-range (eg. 1024-65536) on your FTP client. To minimise the security vulnerability window in opening up a large number of ports, try to determine the baselines and finetune the port-range.

To rectify this problem, use passive FTP.

Hope this helps. Regards.

Steven Sim
Brainbench MVP for Unix Admin
http://www.brainbench.com

[James A. Donovan]

As an update to my problem....the problem does not appear to be with either my server or the remote ftp server. Rather it appears that sending ANY file larger than 1406 bytes will cause my ftp session to hang. The MTU size on both servers is 1500 as is the MTU size on the switch and the router that exist before my firewall. As I'm led to understand, there is an encrypted VPN tunnel from our firewall to the firewall of the remote ftp server.

Other than determining that all network devices between here and there have an MTU of 1500, what can be done to help solve this problem?

[James A. Donovan]

Thank you everybody who responded. The problem is resolved. I should have seen it earlier. Remember way back when, when HP issued a security warning regarding the ip_pmtu_strategy variable? And that they recommended setting the value to 1? Well, it seems there must be at least one piece of hardware between my site and the ftp server which is dropping packets. Because that setting just ain't working for me.

I set the ip_pmtu_strategy to 2 and everything worked.