FTP Script with succes and fail logs

[syedrafed]

hi techies..

SO here i'm messed up with my script for ftp..
i have to ftp a server 10files from common directory and put some files there and i want the script to create a file named success and the files which have been transferd should generated logs here with date as extension(for all 10 files) and vice versa with file name failed..
lets assume my scenario
DIR=/tmp/pcs/FILES -here my files ex-f1.gz f2.gz....f10gz.
logs shoul be created in same dir as
/tmp/pcs/succes----f1.gz.23052011 tranfered succesfully
/tmp/pcs/failed-----f3.gz.23052011 failed

i hope its clear for you guru's to guide me

Awaiting replies

Thanks and Regards

[Mel Burslan]

SERVER=some.name.com
SOURCE_DIR=/location/of/your/file/on/local/server #make absolutely sure this dir exists
DEST_DIR=/location/on/ftp/server
USER=ftp_server_username_here
PASS=ftp_server_password_here

ftp -n ${SERVER} <user ${USER} ${PASS}
lcd ${SOURCE_DIR}
cd ${DEST_DIR}
prompt
bin
mput *
EOF_EOF

TEMP_DIR=/tmp/somerandomdir
# mkdir /tmp/somerandomdir #if necessary

ftp -n ${SERVER} <user ${USER} ${PASS}
lcd ${TEMP_DIR}
cd ${DEST_DIR}
prompt
bin
mget *
EOF_EOF

#check if they copied successfully
# modify echo statements according to format you need

cd ${SOURCE_DIR}
for i in `ls -1` # this assumes there are no other files in this directory
do
diff ${i} ${TEMP_DIR}/${i} > /dev/null 2>&1
r=${?}
if [ ${r} -ne 0 ]
then
echo "${i} transfer failed"
else
echo "${i} transferred successfully"
fi
done >> logfile

rm ${TEMP_DIR}/* # clean up temp files



Hope this helps. I have not tested this but the logic is quite clear. upload them to ftp server, download just uploaded files into a temporary location, Compare original and the downloaded copy. If there is any difference between the two, transfer was not successful. Not bullet proof but should work most of the time

[Bob_Vance]

Here's a start for using FTP in a script.

The first 'ftp' lists the files that are actually on the server.
It creates a log of the list in ".iflist" .

The second 'ftp' attempts the actual download and creates its log in ".glist" .

This example uses a file pattern to download as in your example.
There's no provision checking for individual files, although that could be added.



#
cd ~/tmp/pcs
FTPpatch_site=ftp.itrc.hp.com
user=anonymous
upass=a@b.c
FileDir=/product_patches/hpom/hpos/scripts/scripts_for_4.0
Files_to_get="*.pl"
FLIST=.iflist # file list found on ftp site
GLIST=.glist # download log

touch .new

# get list of files that actually exist
ftp -nv <<-!EOF > ${FLIST}
open $FTPpatch_site
user $user $upass
binary
prompt
cd product_patches/hpom/hpos/scripts/scripts_for_4.0
pwd
ls "$Files_to_get"
quit
!EOF

# try to download the files
ftp -nv <<-!EOF > ${GLIST}
open $FTPpatch_site
user $user $upass
binary
prompt
cd $FileDir
pwd
mget *.pl
quit
!EOF



The script could then process/compare/manipulate the ".iflist" & ".glist" logs to produce the "success" and "fail" logs that you want.
Then these could be similary uploaded (ftp put) to the server.


For example,

this gets the files' listing from the log:
grep < ${FLIST} -e ^-

this gets the simple name from the listing, assuming no blanks in names:
grep < ${FLIST} -e ^- \
|awk '{print $NF}'


This would be a cursory attempt at a success log, but,
there's no file size checking or anything,
just a "Transfer complete" :

grep < ${FLIST} -e ^- \
| awk '{print $NF}' \
| while read attempted_file
do
if grep " $attempted_file " < ${GLIST} ; then
awk < ${GLIST} '
/Opening.*'" $attempted_file"' /{
getline;
if (match($0,"Transfer complete")) {
print;
getline; print;
}
exit
}
'
else
echo $attempted_file: not transferred correctly
fi
done




bv

[klb]

Nice, but what if the files are huge and take hours to move around?

I was thinking more along these lines:

TMPFILE1=/tmp/myftptmp_$$

ftp -v -n <<-EOT > $TMPFILE1 2>&1
open HOST
user UID PASS
cd
mput *
bye
EOT

status=$?
# Check $status for non zero ...and...
# Now, just go and interrogate $TMPFILE1
# for signs of failure ( watch for
# various ftp codes and error messages )
# and report or touch files as necessary
# to notify status

rm -rf $TMPFILE1

[Bob_Vance]

I'm not sure that ftp returns a meaningful status.


bv

[Mel Burslan]

ftp almost always returns a 0 (zero) exit code and short of concocting a poor man's version of ConnectDirect (formerly known as NDM) with server and client side agents to perform checksum calculations, only way to assure successful transfer is to upload and then download it for comparison. you can do an remote 'ls' and compare the byte size of the files but it is not a guaranteed verification in my opinion.

my 2 cents

[Mike Miller_8]

Create a cksum file of the files to be sent.
Send that file with the files over to the other server.

Have a cron job on the other server run periodically and look for the cksum file. If it exists, run a cksum on the files that are there (minus the cksum file sent) and compare the checksums. If they do not match, kick out an error msg. If they do, rename the cksum file and add text indicating that the transfer was successful.

[James R. Ferguson]

Hi:

Checking the return code (exit status) from an FTP is virtually meaningless.

FTP gives a 3-digit response for every action requested of it. Have a look at the 'ftpd(1M)' manpages for what these mean. Together with this response code comes a message that can be used to discern success or failure if you write ('-v') the session's messages to a file. Do *not* rely on the exact verbiage as it will differ when interacting with different server operating systems.

However, you can easily parse the session log after an FTP transfer to determine success or failure.

Comparing check-sums of transferred files is only guaranteed to work if the transfer is a binary one. If you move files between platforms that have different record delimiters this will fail every time.

Consider that Windows servers use a CRLF (carriage-return + line-feed) as a "line" or record delimiter. Unix-based systems use a bare line-feed. Hence check-summing a file transferred between these dissimilar architectures is doomed to differ.

Now that we have offered lots of advice, please read and follow this:

It's a nice way of saying "thanks".

Regards!

...JRF...

[klb]

I check the return code always regardless, for all unix scripting commands that return status.

It was suggested to check that AND look for messages in the FTP output that indicate success/failure.

In my scripting around ftp, I use the return codes inside the log file to determine whether a given transfer was successful.

That is what I suggested the OP do.

Thanks,

-klb