ftp not completing all files

[James Ellis_1]

Trying to FTP several files from one server to another. The load process starts and moves 100 flat files to my server, and then loaded to an oracle instance. The load process stops about half way through on a 33mb source file that loads without data, and the rest of the files are loaded without data. This happens every time, and stops at the same 33 mb source file. Note that the previous FTP files were large files also and successfully loaded. When they tried to load the files manually, it worked. I've checked some preliminary things, but my ndd on /dev/ip reports a 1 while another server reports a 2. My mtu on the server is 1500 and the router is 1534 I believe. Also, the total load process time used to be 15 minutes and now its taking up to 1 hour and 15 minutes. Any suggestion on what to look into? Thanks.

[Martin Johnson]

have you compared the file sizes and checksums between the source and the target to ensure the ftp copy completed correctly?

HTH
Marty

[Steve Post]

two things to check:
I had a problem with very long ftp's between two hp9000's. The problem was with the speed setting on the lan card conflicting with the network switch. But you run it by hand and it seems to work. So this is probably not it.

Another idea: your environment when running it via cron is slightly different when running it by hand? Perhaps you are a different user? Or variables used for the ftp are not set when running via cron?
Type " env > /tmp/run.byhand"

Add " env > /tmp/run.bycron"
to your cron script.
Now you can compare the two text files.

just trying to help.
steve

[harry d brown jr]

James,

On the receiving end, what are the permissions of the file that it's hanging on?

Also, think about rewriting your ftp script to use perl, if you already haven't:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xae627bb04b5cd611abdb0090277a778c,00.html

it allows for better error reporting, trapping, and restarting.

Also, it would be better if you tar'ed those files up into a single file and ftp'ed them.

live free or die
harry

[A. Clay Stephenson]

I would first check the usual suspects. I assume that when you compare the 75 minutes loads to what were once 15 minute loads that this is a fair comparison. i.e. data volumes are comparable; machine loads are similar; network topography is the same; ... . If that is the case the the very first thing to check would be the speed/duplex settings on both hosts and on the ethernet switch ports. I strongly urge you to hard set those on both ends. If one end fell back to 10MB, ftp will continue to just barely work and telnets will appear normal.

I also would make sure that you explicitly set the FTP transfer mode as needed.

It seems to me that after you get any network problems resolved, you still have a fundamental problem. You are assuming that your FTP scripts work. A better idea is to introduce some error checking and an automatic retry if you receive a bad status. The easy way to do this is via Perl with the Net::FTP module you can download from www.cpan.org. That will make the error checking about as easy as it gets.

The attached perl script will get you started and will work on UNIX, Windows, VMS, ...

Use it like:
ftp.pl myfile1 ...
STAT=$?

if [ ${STAT} -eq 0 ]
then
echo "Transfer of myfile1 ok"
fi


This example, logs in, cd to /tmp, sets up for ASCII transfers (you can simply change 'acsii()' to 'binary()' if you need to), and starts tranferrring myfile1. It looks at the transfer status and will retry up to 5 times.

[James Ellis_1]

Martin,
No, have not checked the two files with checksum. The problem is in the directory where the ftp files are going, I show approximately half of the files loaded with data and the others come with "0" data. I will ask the person who is running this load process to use checksum to make sure he got it all FTPed, however, we are still not completing the load process yet. thanks.

[James Ellis_1]

Steve,
I am not sure how the user is running their load process. It may be by a cron job, but its being ran from another server, which I believe it a MS server. Thanks.

[Mark Greene_1]

Try doing the ftp with the "hash" command active. This will send packets back to the oringination server and not let the tcp connection timeout, if that is what is happening.

Also, check the obvious stuff: make sure you have enough space on the target server, *and* enough inodes, make sure that the nics on both systems are set in coordination with the router to which they are attached (either 100 half or 100 full, but not auto), make sure quoatas haven't been enabled on directory on the target server, and if you are using ACL's that you aren't having a problem there.

HTH
mark

[James Ellis_1]

Harry,
The permission on the file that is hanging on is the same as the previous successful FTP files. However, the owner/group of the ftp_dump directory is itapload/usr and I go into the ftp_dump directory, I see the owner/group of all the files are itapload/dba? I doubt this is the problem, because the other files prior to the hand were successful. Thanks.

[James Ellis_1]

I went into Sam, and cicked on the "Networking and Communication" icon.

Under this is a listing, and the FTP is listed as Disabled.

But users have always been able to FTP, so what is this Disabled supposed to disable?

Thanks.

[A. Clay Stephenson]

Disabling FTP there means than ftpd does not run. Your box cannot be an FTP server but it can still be an FTP client.

[Systeemingenieurs Infoc]

James,

check if there's enough diskspace in the directory containing the ftp-files.

Go to the directory and execute 'bdf'

hecou