FTP-Script with many mget-command hangs

[Volker Borowski]

Hi folks,

I have a problem with a ftp-script, I need to run frequently. Script goes:

user ftp ftp
prompt
binary
mget dir1/dir2/*.txt
get dir1/dir2/picture.jpg
: repeat this 7000 times for several dirs
mget dir7/dir9/*.txt
get dir7/dir9/picture.jpg
bye

Now this script runs 3 out of 20 times ok.
Mostly it suddenly hangs, keeping an established Port 20 and 21 connection showing up in "netstat -n" (and sometimes it keeps even two port 20 connections).

Setting the Script to some 20 entries usally works at a rate of 99.9xx percent (which keeps me thinking, that firewall-rules are not the problem).

What strikes me is, that each mget/get opens a seperate data stream, so I suspect something is blocking in the connection table, or in the socket administration.

Any idea, if I need to adjust kernel (which value) to be able to handle more sockets.
Can I set additional parameters inside ftp ?

BTW: I am facing the same problem with an ftp-client runnning on HP-UX, Solaris and Linux.

Thank you
Volker

[Craig Gilmore]

My first question is: Why is ftp being used instead of rcp? I'm guessing that the only reason is that the system being copied from does not support it.

If this is the case, then I think I would write a socket program that would be used instead of FTP. Why? There is no way to be sure that the files copied with the mget is successful.

Unfortunately, there is no kernel parameter that you can change for this part of sockets. You can use nettune on 10.20 and ndd on 11.00 to set the number of dynamically allocated sockets. On 10.20, tuned high, and on 11.00 this range is 49152-65534. 20000 sockets should be enough for dynamic use to copy 7000 sockets. You might be running out of system memory however.

I recommend re-thinking the use of ftp. It is not designed for the type of work that you are trying to do.

[Volker Borowski]

Hello Craig,

unfortunately, this is a public service I use, and the FTP-Server is not mine :(
No way to change it.

Volker

[Brian Markus]

This may be considered "tabu", haha but have you considered useing a "ftp bounce" client on a winblows box? I use a program named flashFXP, wich opens up a connection to server A and server B, then willl transfer between the two.
The app allows draggin and dropping, and mass selection. You can setup queue's in the program. Everything is very automated and easy to use.

Just a thought.

Brian.

[Craig Gilmore]

OK, since we are working with a public domain server... it makes this harder.

Not knowing what the other side of the connection is and how it is setup also complicates the issue. It is possible with wftpd to limit the number of accesses. If the server is wftpd or some variant, then it should state this on connection. Check first for this.

Next, if not wftpd or unknown, then I would suggest doing a nettl trace or a tcpdump trace of the network traffic. Are the commands issued from the HP successfully? Is it the side that you can control or the server side that seems to be where the problem is. If you find from the trace that the problem is in the server, then you have reached a point of negotiation with the service provider... or a brick wall.

If the problem is in the HP then we can dig further.

Regards

[Volker Borowski]

Hello Craig,
the remote site is Microsoft FTP-Server 4.0 !
I checked man page for tcpdump....
... nothing you use so frequently.

Any tip, where to begin with ?
Thanx
Volker

[Craig Gilmore]

OK, Windows box is the server.

Starting questions:
1. Are other ftp's requested from the same server able to compelete successfully at the same time that your mget script is running?
- If so, this probably means that there is something specific about the mgets that may be causing the hiccup.

2. Have we checked with the server to see if any other users have reported a problem? This may tell us if we are breaking new ground or it is something that they are already working on.

-------

Tracing: tcpdump is a public domain tool that many people use. HP has their own tracing tool, nettl, that is built into the OS. tcpdump has advantages and problems as does nettl. If you don't already have tcpdump installed and familiar with it's use, I recommend saving time and using nettl.

1. You will want to be able to re-create the problem.
2. You turn on nettl, recreate the problem, and turn off the trace. nettl uses circular capture files. You need to turn off tracing soon after reaching the problem to be sure that you capture the info in the trace files.
3. To turn on tracing, use:
#nettl -tn pduin pduout -e ns_ls_ip -f (capturefilename) -tm 99999 -m 256

What this means is: -tn turn on tracing
-e entity or level of tracing. In this case at the IP level.
-f filename there will be .TRC0 and/or .TRC1 appended to this name. file will be created in directory you issue the command from, unless you give absolute path.
-tm This is setting the tracefile to it's largest size.
-m this is how much of each message to capture. In this case the first 256 bytes of the message.

4. recreate the problem.
5. Turn off tracing: #nettl -tf -e all
6. Use netfmt to format the data and look for the conversation. USE FILTERS !

A filter file, named xxy, could contain:

filter ip_saddr (ip of the windows server)
filter ip_daddr (ip of the windows server)

Using this filter file with netfmt would look like:

#netfmt -Nl -c xxy -f filename.TRC0 > fmtfile

or

#netfmt -1 -c xxy -f filename.TRC0 > fmtfile

This will take some time, but it may help answer why the conversation is failing.