Make sure the file transfer is suscess

[ust3]

I hv two servers ( server A and server B ) and have a script to run to ftp files (may be more than one file) from server A to server B in every 15 minutes , I would like to make sure the files are transferred successfully to other server , can advise what is the best way to check the files are successfully transferred or not ? if not successful , then send me a notifiy mail , can anyone provide the script or give a suggestion for it ?

Thx in advance.

[Oviwan]

Hey

Create the checksum of both files and compare them.

chka=$(cksum fileservera)
chkb=$(cksum fileserverb) #here of the remote file, you can use remsh..

if [[ "${chkb}" != "${chkb} ]]
then
echo "copy failed"
fi

Hope this helps

regards

[rajdev]

Hi,

You may write a script to do ls -l of the file and get the file size, then you can take the ftp output to see how many bytes are transferred , compare them and you should be able to see if its successfully transferred.
The ftp output is the easiest way you can say that file is transferred or not.

see the below for details
---------------------------------------------
# ftp test.com
Connected to test.com
220 test.com FTP server (Revision 1.1 Version wuftpd-2.6.1(PHNE_32286) Fri Apr 15 17:37:21 GMT 2005) ready.
Name (test.com:root):
331 Password required for root.
Password:
230-No directory! Logging in with home=/
230 User root logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /tmp
250 CWD command successful.
ftp> !ls -l qq5
-rw-rw-rw- 1 root sys 4831 Nov 4 03:17 qq5
ftp> put qq5
200 PORT command successful.
150 Opening BINARY mode data connection for qq5.
226 Transfer complete.
4831 bytes sent in 0.00 seconds (15267.88 Kbytes/s)
ftp> quit
221-You have transferred 4831 bytes in 1 files.
221-Total traffic for this session was 5342 bytes in 1 transfers.
221-Thank you for using the FTP service on test.com.
221 Goodbye.
---------------------------------------------

Regards,
RD

[ust3]

thx replies,

Oviwan's method seems quite simple , but I am not too familiar with script writing , can advise how to compare the chka with the chkb in remote site ? except remsh ( it seems not safe ) , can advise what other remote connection command ( ssh ) can do that ?

thx

[Oviwan]

if you have configured ssh you can use this to create the checksum of the remote file:

chka=$(cksum fileservera | awk '{print $1}')
chkb=$(ssh serverb cksum fileserverb | awk '{print $1}'

now compare the two checksums.

[A. Clay Stephenson]

If you are going to use FTP to transfer files then it's dumb to try to shell script it because error checking is so tedious. Use Perl's Net::FTP module and you get error checking for free. The attached Perl script will do everything for you and all you have to do is check the return status. If it's zero then all is well --- just like a real UNIX command (and you don't have to know any Perl).

#!/usr/bin/sh

typeset -i STAT=0
perl ftpput.pl -h mouse -l mickey -p secret -B -d /aaa/bbb file1 file2
STAT=${?}
if [[ ${STAT} -ne 0 ]]
then
echo "Transfer failed; status ${STAT}." >&2
fi
exit ${STAT}

This will login to host "mouse" as user "mickey", password "secret", set binary mode, cd to /aaa/bbb, and then put file1 and file2. If any of these operations fail, a non-zero return status is set. If you use a .netrc file, you don't have to use the -p password option --- just like real ftp.
Invoke as ftpput.pl -u for full usage.

[A. Clay Stephenson]

... and should you wish to turn the world around and instead do gets, here's ftpget.pl. Invoke as ftpget.pl -u for full usage.

[ust3]

thx replies,

I am not familiar with perl script , can advise other script like ksh shell script ? thx

[Patrick Wallek]

Clay's scripts are pretty straightforward if you look at them closely.

I see no reason not to use them. Dismissing something because you are not familiar with it is not good system administration. Part of your job is to continue learning and this is a prime opportunity for you to do just that.

Go back and read Clay's post, and look at the script. You should be able to figure it out and use it pretty quickly.

[ust3]

thx replies,

because the remote site is not in our office , we can't install module to it .
I hv another suggestion , how about get back the transferred files ( by get command ) from remote server to local server , then compared it locally , ( please ignore the traffic issue ) , I think it is a way stupid but very accurate , can advise how to do it ? thx

[ust3]

thx all replies,

I would like to correct my question , the remote host ( server B ) is windows server ( I have no permission to install anything to it ) , so I think ssh is not work, can advise another way that I can confirm the file transfer is totally success ?

Thx in advance.

[Yogeeraj_1]

hi,

Steps:
1. Ftp the file from source to dest
source: file1.dat
dest: file1.dat

2. Rename file
source: file1.dat
dest: file20071123.dat

3. verify the check of file at source (CKSUM1=$(cksum file1.dat | awk '{print $1}')) with checksum of file dest (CKSUM2=$(cksum file20071123.dat | awk '{print $1}')).

4. If CKSUM1 = CKSUM2 then file transfer is OK, else delete files at dest and retry operation.


If you need further help, please let us know.

kind regards
yogeeraj

[A. Clay Stephenson]

The point is to use the Perl script I attached YOU DON'T HAVE TO KNOW ANY PERL. You can treat the script as a black box as call it from within a shell script. If the Perl script returns a zero exit status, all went well. If anything goes wrong at any step, a non-zero status is returned and an error message is printed on stderr --- just like a well-written UNIX command.

[ust3]

thx replies,

I have tried windows is not support "cksum" , is the perl module supported on my case ( local server is linux , remote server is windows ) ? please advise ? thx

[Yogeeraj_1]

hi,

> is the perl module supported on my case?

Since your local server is linux,
and remote server is windows

You do not need to worry about cksum not being available on your windows host.

to check if the perl module is installed, please run the following commands:

#check if perl is available
which perl

#check perl version
perl -v

#check version of module Net::FTP
perl -MNet::FTP -e'print "$Net::FTP::VERSION\n";'

hope this helps!

good luck

kind regards
yogeeraj

[ust3]

thx reply ,

"Since your local server is linux,
and remote server is windows
You do not need to worry about cksum not being available on your windows host."

but when I run cksum on windows , it pops " 'CKSUM' is not recognized as an internal or external command , operable program or batch file" .

do you mean I can't use cksum on wondows ?

I think the FTP module is installed on my system .

perl -MNet::FTP -e'print "$Net::FTP::VERSION\n";'
2.56

but I am sorry I still not know how to use it .

[ust3]

on the other hand , even I can use cksum on windows, how can I use ssh / remsh to login windows ( the windows default is not included these services ) ? thx .

[Arturo Galbiati]

Hi,
since your target server is Windows an intermediate step is needed:

1.cksum file # on source server
2.ftp put file to target server
I suggest you to use Clay's Perl script to improve the error check
3. ftp get file from target server
(see a.m. comment in step 3)
4. cksum file # on source server
5. check if chsum are equal or not

HTH,
Art

[James R. Ferguson]

Hi:

The problem with relying on a checksum or a character count when you are transfering files between different operating systems is that ASCII FTP transfers will add or remove carriage-returns to surrounding newline characters in your text file. This will means differences between checksums and character counts on disparte platform types.

As noted, one way to circumvent this, when transferring between platforms that describe line endings differently, and to rely on character counts and checksums to validate your transfer, is to FTP the file from the local host to the remote one and then immediately FTP the copied file back. Comparison of the original and the imported copy can then be made. Of course, this *doubles* your network overhead.

A Perl solution as offered by Clay avoids all of this. One other alternative is to verbosely log the FTP session's messages into a temporary file. When the file transfer has completed, parse the log file to determine if the transfer was truly successful or not.

Your original question ended by asking, "...can anyone provide the scritpt or give a suggestion for it?" You now have both answers. Clay has offered a blackbox script. Otherwise you can craft your own using the suggestions, above.

Regards!

...JRF...