Incomplete file received via FTP

[Michael Treacy]

? Experiencing trouble with files sent from our IBM DOS/VSE box to HP/UX 11.0 via FTP. Lately the file is getting truncated and I need a way to test for a complete file.

Is there a way to test for an EOF marker in a script before I process the file ? Or perhaps count the number of records and compare before processing ?.
The files are relatively small 1200 - 2000 recs.

[Steven E. Protter]

You could do a checksum on the file at both ends of the connection. First on the ibm box, and then afteward on the HP box.

Another possible answer would be to first transfer the file to a PC at the other end. Then you can use a gui ftp tool like ws_ftp and see the progress as it uploads onto the hp box.

If it fails, ws_ftp makes a nasty noise to let you know.

I think the more important issue is to figure out why its failing.

On hp box.

inetd -l

tail -f /var/adm/syslog/syslog.log

to the transfer and watch the errors....

You might get something useful for diagnosis....

SEP

[Zigor Buruaga]

Hi,

In addition, one possible reason for a file getting truncated is a timeout. Check that in the IBM side.

Kind regards,
Zigor

[Michael Treacy]

Thanks for the ideas...
Checksum sounds like it's worth further investigation on my part... Even though I can't find a man entry on it.

The syslog looks like this:
ftpd[8210]: USER hquest
ftpd[8210]: PASS password
ftpd[8210]: FTP LOGIN FROM 10.4.2.12 [10.4.2.12], hquest
ftpd[8210]: CWD /lawp/interfaces/hbo/hc/
ftpd[8210]: PWD
ftpd[8210]: PORT
ftpd[8210]: RETR hqglchg.dat
ftpd[8210]: QUIT
ftpd[8210]: FTP session closed

[Mark Grant]

The way to test for an EOF is to read to the bottom of the file. I realise this isn't very useful but any file you can read the end of, in a shell script, is going to have an EOF marker. It might not be where you really wanted it, but it will be there.

I think SEP's checksum solution is the only way to go here, apart from finding out what is causinmg the problem in the first place.

[Michael Treacy]

man cksum, i'll see if the mainframe boys can send it for comparison.

[Dave La Mar]

Michael -
We do hundreds of these daily From DOS/VSE to HP-UX. I have to admit I have not seen your truncation problem without the DOS/VSE ftp batch job failing.
Couple of things to check:
1. As mentioned, the timeout on your DOS/VSE ftp product. Some use CSI's stack, some use BSI's stack. Those are the two heavy hitters. If you are using BSI's, I can help here.
2. Check the VSE listing from the ftp batch job. Look for byte count as well as errors.
3. Check the timeout option on your UNIX side.
4. As mentioned, if there is an "eye catcher" for the last record in the file, tail the file, grep for your "eye catcher", process or not process on the results.

Best of luck.

Regards,
dl

[A. Clay Stephenson]

One of the best ways to script FTP transfers is the Net::FTP module of Perl. This module is now normally distributed with Perl 5.6.1 and greater but in any case you can download it from www.perl.org/CPAN. Using Perl, it is trivially easy to check the status of any FTP command.

ftpput.pl file1 file2 file2

will login, cd to /tmp, go to ascii mode, and then start transfering file1, file2, and file3. Error checking is done and it will attempt 5 retries before giving up. An exit status of zero is returned on success.

You can avoid the plaintext passwd by using .netrc. I made those comments in the code.

The good news is that this script will run unchanged on any platform that has Perl.

[Ron Kinner]

We had a similar problem with FTPs thinking they were done but with truncated files as the result. Problem was solved by a new NIC card. Seems the old builtin one was EMI sensitive ( a known problem ) and would sometimes drop the ball without telling anyone.

A good test for a NIC is to get your local Cisco guy to do an extended ping of your IP address and tell it to sweep a range of sizes from the smallest up to 18000. He should see nothing but !'s. If you get a . once in a while then the NIC is probably at fault. If you get Q's you need to turn off Source Quench in ndd. Also check that you have the latest patches for the NIC driver. Some of the outdated drivers will cause the same problem.

You might also want to investigate potential bottlenecks in the network and see if they are overloaded and dropping packets. You might have to put some custom queuing on the routers to give your ftp traffic a bit higher priority.

Ron

[Colin Topliss]

Ooops. Joined the party a bit late on this one.

For what its worth, we always used to insist that the files we were to receive had at least a header record that told us how many records were in the file, and more often than not we had a trailor record too. That way we could do some post-FTP validation to ensure a complete file had been transferred. The transfer process itself was done using Perl (so much better than any other method we could find).

If you are able to influence how the files are generated, perhaps you can request that header/trailor records are added?