A different FTP failure

[Ron Kinner]

I have 5 HP 9000/715 running 10.01 or 10.00. Each runs an automatic process which does nothing more than move files which appear in an input folder to another folder then it ftps the files to a 10.20 server (different box from the one which I complained about in my other post) and then finally moves the file to an archive folder. This process has worked fine for 5 years. Suddenly two of my boxes are not emptying their folders. netstat -a shows the FTP session is hanging.
tcp 93 0 CBI-BRA.4618 192.168.112.2.ftp established
which eventually changes to CLOSE_WAIT and stays there. The 93 is supposedly in the input Que so what is it waiting for?

When I do a manual FTP to the same server and mput a bunch of files I get a failure about once every 6-10 files. The failure occurs as the data connection is being established and says No route to host. The manual session continues with no problems but I suspect my application uses a simple put since it has to archive each file after a successful send and that it doesn't understand what is happening. No - I do not have access to the source code - that would be too easy. The other three boxes have not reported any problems but they are not as heavily loaded as the two problem boxes and they may also be 10.20 machines. The boxes are in another country so upgrades are a bit of a hassle. Also makes it difficult to put a sniffer on it.

Just pointed the application to a different 10.20 box. One which did not give me any "No route to host" errors during large file transfers and it hung too. Woe is me. I did so want to prove it was the 10.20 machine so that I wouldn't have to worry about it any more.

We have a new homegrown application in development to replace the current one but it won't be ready until early next year. If there were some way to automatically drop these hung connection the application would probably continue on its merry way. Maybe I could run a Chron job or something to limp through the holidays. I have no desire to fly down there and reload them.

Any help either solving the problem or kludging it to keep it working would be appreciated.

For what it is worth we are working over a sat link but it appears to be error free.

Ron

[Steven Sim Kok Leong]

Hi,

With regards to "No route to host",

1) What is your routing table with netstat -rn? Are there multiple default routes?

2) Are there any packet losses with ping to the router over the length of time of your FTP transfer?

3) Does it occur only for FTP? To isolate whether the fault lies with FTP, install SFTP2 (comes with SSH2) and try SFTP'ing the same files. If installing SSH2 is too troublesome, enable RCP. If RCP works, then the fault is application-related. If RCP also fails, the fault lies elsewhere below the application layer, ie. TCP/IP could be problematic etc.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Ron Kinner]

I find this hard to believe but have just been informed by the operator that maximizing the window in which the process runs causes the application to process a couple of files before stopping. He then minimizes the window and then maximizes it again and it does a few more files. In this way he is able to "pump" all of the files through the system.

Ron

[Kofi ARTHIABAH]

Ron:

You can actually get some sniffing going on on he boxes by running HP's nettl (do a man on this command).

Also, you might want to use ndd to find out what your ip_pmtu_strategy is set to:
for HP 11 - use:
# ndd -get /dev/ip ip_pmtu_strategy

you would want to set it to 0 with
# ndd -set /dev/ip ip_pmtu_strategy 0
(this disables path MTU discovery)

for 10.20 you have to use the nettune command - and the parameter is tcp_pmtu:

# nettune -l tcp_pmtu (to display current value)

# nettune -s tcp_pmtu 0 (to turn off mtu discovery)

I had a similar problem which turned out to be because one server was set to discover the MTU size but the intervening networking gear were not responding correctly, so turning off the MTU discovery fixed the problem.

Good luck

[Ron Kinner]

Sorry I'm slow getting back to you. Somehow the "notify me" option got unset and I didn't know anyone had replied.

Answers to Steven's Questions:

netstat -rn on the client machine shows only a single NIC and a single default route which goes to my router. (I'm really the network admin but as part of my job I also maintain these client machines.)

The server has a static route to the client.

The funny thing is this works most of the time and nothing is changing in the routing.

Pings work all of the time though we are working over a sat link (and then a cross country T1) so our round trip delay is on the order of 750 ms.

I can't try other protocols since I have no login on the server.

I wrote a script program which backs up the automatic transfer by opening an FTP session every 30 minutes. Its log files reveal three types of failures:
1. The no route to host problem previously mentioned.
"200 PORT command successful.
425 Can't build data connection: No route to host. " Turns out this one is fairly common but it comes too often to be the cause of our problem.

2. The session is established and then we never hear from the server again.
"Connected to camc.
421 Timeout (900 seconds): closing control connection.
Not connected.
Not connected. "
This one is probably the one which causes the trouble with the other program. It happens just often enough to be the culprit. Netstat -a shows an ftp connection Established with a constant value in the in queue. Eventually goes to FIN_WAIT but never drops completely .

3. There is no response from the server. This one is really rare. Can't find it in the log now but I know I saw it once. I expect it is probably just rain fade. This system has operated for 5 years and as one of the boxes is in Brazil I don't think this bothers the program much. At least it wouldn't leave an open ftp session.

Answers to Kofi,

Never been able to get nettl to run. Will have to try it again. Some of the urgency is now gone since my little workaround is hiding the problem.

tcp_pmtu is set at 1. I will try turning it off but it seems unlikely. The other three which work all take the same path to the server except for the last link and that is over identical cisco 7000 routers which have essentially identical configurations.

Thanks

Ron