ftp failures

[John Waller]

Hi,
I have an interesting problem in that an automatic FTP script occasionally (12 times out of 50,000 over last 2 days) appears to loose the data packet of an ftp transfer.
We have an application which creates small files between 400 and 500 bytes and ftp's them to a remote machine across a WAN. Occasionally these files are received with a size of 0 though we can verify that they were created with a non 0 size on the local machine. Has anybody seen this before or can explain how it can occur. I have tried everything I know to manually re-create the situation without success. We ftp and use individual "put" commands to place the files onto the remote system, also we are not getting any indication of failure in the syslog.log.

[Michael Schulte zur Sur]

Hi,

can you rule out, that files were transmitted, while there werent finished set, just created with size zero?

greetings,

Michael

[John Waller]

The files are created in a temporary directory then we use mv to move them to the transfer directory before initiating the transfer. As we do not issue the mv in the script until the file creation has completed we can be certain the file was non-zero before transfer.

[Jean-Louis Phelix]

Hi,

You are sure that file has been created before the 'mv', but are you sure that transfer is not initialized *before* the 'mv' ends ? Files are small, but if your IOs are freezed for a very short time ... Of course this can occur only if source and target directories are not in the same filesystem (in this case, mv = cp + rm). You could transfer the file by mv to the target directory using a leading dot in the name, then mv the file *in the same filesystem* to remove the leading dot which would be immediate in this case.

Regards.

[John Waller]

The source and target for the mv are in the same filesystem. We move from /u/transfer/outbound/tmp to ../ (i.e /u/transfer/outbound)

[John Palmer]

Does the ftp process itself display any sort of error message?

[John Waller]

Looking at the script it appears due to the large number of transfers the person who wrote the script directed 2> (i.e standard error) to /dev/null. I have put a request to have this directed to a file so we can see any errors. One problem when you have non system admin aware programmers writing ftp scripts.

[Bill Hassell]

You'll also want to turn on detailed ftp logging in /etc/inetd.conf but beware that syslog will grow rapidly with all the options turned on:

-l logs all sessions
-L logs all commands

and in the /var/adm/syslog/xferlog file:


-i logs all files received
-o logs all files sent

xferlog may be the most useful (see man xferlog) along with adding -v to the ftp command in your script and saving both stdout as well as stderr from ftp.

[rick jones]

If there are transport or network-level problems with the file transfers, you might be able to establish a corellation bewteen things like connection aborts reported in netstat -p tcp with your "failed" FTP trasnfers.

As for getting rid of the 2> stuff in the script, you could direct the stderr stuff to a tempfile that is deleted if the transfer is determined to be completed successfully.

[Brian Watkins]

We have seen this happen randomly on several systems under very similar circumstances to your own.

After several weeks of continuous packet sniffing and combing through VERY LARGE sniffer logs, we could find no root cause or technical explanation for the 0 byte files occuring.

Even though FTP is TCP-based, and TCP is (in theory, anyway) connection-oriented, packet losses should never occur. In the real world, it does happen from time to time. Since FTP is essentially a "dumb" application and has no built-in error detection/correction routines, packets do occaisionally get dropped resulting in 0 byte files on the target host.

We found that the best way to prevent 0 byte files is to build in a couple of layers of error detection and correction to the FTP scripts themselves that will verify that the target and source files are the same size prior to moving on to the next transfer. If they don't match up, then resend the files.

This probably isn't the answer you were looking for, but after pulling our hair out for 2 and half weeks, our networking and Unix teams decided that error checking and correction was the fastest and easiest solution.

Good luck and Happy New Year!

Brian