topic Re: Automate FTP in Operating System - Linux

Automate FTP

Coolmar — Wed, 09 Nov 2005 15:56:29 GMT

Hi,

I need help with figuring out how to script the following. There is an FTP server that contains a file that I need to download. A new one is put there almost every day...I don't know when, it is random. If there is not a new file put there, then the old one remains until the new arrives (different name each time). I know how to automatically ftp, but want I want to do is:

1. only download the file if it is new (in other words, I don't have it downloaded already)

2. email myself as a notification that a new file has been obtained.

Thanks!

Re: Automate FTP

Rick Garland — Wed, 09 Nov 2005 16:10:09 GMT

An option to automate ftp file transfers without using the $HOME/.netrc file...

ftp -n -v hostname > $LOG << ENDFTP
user username passwd
prompt off
binary
cd wherever
lcd wherever
mget *
quit

If you are wanting to do some comparisons for newer file, will need a script that contains logic to compare the date/time stamp of the file to be downloaded.

Re: Automate FTP

Coolmar — Wed, 09 Nov 2005 16:16:41 GMT

Thanks Rick for the response. I already have the script and it works fine...it is the "logic" that I am looking for.

S.

Re: Automate FTP

Steven E. Protter — Wed, 09 Nov 2005 16:27:06 GMT

Hello,

The ftp client is not very good at knowing whether the file is newer or not.

You could however insert some code into the excellent response you have already received.

ls -la | awk '{print $5 $6 $7 $8} > /tmp/localfile

# now you can compare the time and date stamp and size to what you already have on your local system.

I'm not sure it will be 100% accurate.

A better methodology perhaps would be for the ftp server to run a command like this

find /ftpdir -mtime -exec 'ls -1 > /ftp/downloadlist'

or maybe

find /ftpdir -ctime -exec 'ls -1 > /ftp/downloadlist'

Then you download that list every day, feed it into your ftp scripot and get the new files.

A little help at ftp side is needed.

Or you can download and compare later, but it sounds like you are trying to conserver bandwitch.

Hope I've given you an idea on how to approach the problem.

As for the email part:

echo -e "To: $EMAIL_TO" > $fmesg
echo -e "From: $EMAIL_FROM" >> $fmesg
echo -e "Cc: prottertemp@investmenttool.com" >> $fmesg
echo -e "Subject: $EMAIL_SUBJECT" >> $fmesg
echo -e "" >> $fmesg
/bin/cat $fbody >> $fmesg
cat $fmesg | /usr/sbin/sendmail -t

> $fbody
rm -f $fbody
> $fmesg
rm -f $fmesg
> $ftext
rm -f $ftext

just set the variables and decide what the text is. fmesg is a temporary holding file for the message. Should be pretty obvious.

This is production code from a Linux box, but it is also running on HP boxes I used to admin in the US. Linux boxes are mine, I'm allowed to peak.

Good Luck,

SEP

Re: Automate FTP

Dave La Mar — Wed, 09 Nov 2005 16:32:05 GMT

Sally -
There are probably more eloquent ways to do this, but ...

From and ftp session:
1. Set the prompt off by issuing the ftp prompt command.
2. ls yourfile* dummy
The above will list yourfile with date/timestamp in the file dummy on your local machine.
3. Quit the ftp.
4. Compare your old file to that in the dummy file created in #2.
5. Based on the comparison, go on with your business. i.e. set off the email and second ftp job to pick up the file.

Best of luck.

Regards,

dl

Re: Automate FTP

James R. Ferguson — Wed, 09 Nov 2005 16:34:14 GMT

Hi Sally:

See the man pages for 'ftp(1)'. You have a 'newer' variation of 'get':

# newer filename

...gets the file only if the modification time of the remote file is more recent that the file on the current system.

Regards!

...JRF...

Re: Automate FTP

Rick Garland — Wed, 09 Nov 2005 16:36:20 GMT

Can use the 'ftp newer '

Get the file only if the modification time of the remote file is
more recent that the file on the current system. If the file
does not exist on the current system, the remote file is
considered newer. Otherwise, this command is identical to get.

As to sending mail, can have a job that periodically checks the date/time of the file. If newer, then send mail.

DATE=`ll | awk '{if (($7 == "9")) print $9}'`
if [ $DATE != date +%e` ]
then
mailx ....
fi

Hope that helps.

Re: Automate FTP

Rick Garland — Wed, 09 Nov 2005 16:37:39 GMT

BTW, I use the $7 == "9" as an example. Today's date being 9

Re: Automate FTP

A. Clay Stephenson — Wed, 09 Nov 2005 17:06:07 GMT

Actually the FTP is extremely good at knowing whether a file is newer or not iff you use the Net::FTP Perl client but that should not be necessary if I can depend upon you being truthful about the filename changing everytime. All we have to do is list the remote directory and compare those names to the one found in the destination directory on the local machine. Any files not found should be retrieved and mail should be sent. I already have a Perl script which will do all of the hard work, we just need to add some shell logic.

ftp.sh
--------------------------------
#!/usr/bin/sh

PATH=${PATH}:/usr/local/bin
export PATH

typeset PROG=${0##*/}

typeset TDIR=${TMPDIR:-/var/tmp}
typeset PID=${?}
typeset T1=${TDIR}/X${PID}_1.txt

trap 'eval rm -f ${T1}' 0 1 2 15

typeset SRCDIR="/home/cstephen"
typeset REMHOST="bugs"
typeset DESTDIR="/tmp/cstephen"
typeset U="cstephen"
typeset P="-p secret"
typeset ADDRESS="mmouse@disney.com"

typeset -i STAT=0

# make sure destination directory exists"

if [[ ! -d "${DESTDIR}" ]]
then
echo "Destination ${DESTDIR} does not exist." >&2
exit 1
fi

cd "${DESTDIR}"
STAT=${?}
if [[ ${STAT} -ne 0 ]]
then
echo "Cannot cd to ${DESTDIR}; status ${STAT}." >&2
exit 2
fi

#Get a list of files in directory ${DIR} on remotehost {REMHOST}
#Login as user ${U} using password ${P}.
#If a .netrc file is located on the client then no passwd is needed.

ftpget.pl -h ${REMHOST} -l ${U} ${P} -L '' -d "${SRCDIR}" > ${T1}
STAT=${?}
if [[ ${STAT} -eq 0 ]] # dirlist okay
then
if [[ -s ${T1} ]]
then
typeset FNAME=""
cat ${T1} | while read FNAME
do
if [[ ! -f ${FNAME} ]]
then
ftpget.pl -h ${REMHOST} -l ${U} ${P} -d "${SRCDIR}" "${FNAME}"
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then
echo "New File: ${FNAME}" | mailx -s "New File" ${ADDRESS}
else
echo "Failed to get ${FNAME}; status ${STAT}" >&2
exit ${STAT}
fi
fi
done
fi
else
echo "${PROG} dirlist failed; status ${STAT}" >&2
fi
exit ${STAT}

-------------------------------
That should do it if I haven't made any typo's. You can add a patterm to the ftpget.pl -L argument to match only certain patterns. e.g -L 'Data*.txt'. Using Perl is a much better choice because you get error checking for free -- something that is very tedious in the shell. Note that you can also setup a .netrc file on your end and you don't need to pass the password. Man .netrc for details. If you want to do this more secure there is also a Net::SFTP module. You can then make a cron wrapper script for this and you are done.

Here's the perl piece, ftpget.pl; the above script assumes that you install it in /usr/local/bin. I suggest that you invoke it as ftpget.pl -u for full usage because you may want to set binary or ASCII mode for the gets.

Re: Automate FTP

Dave La Mar — Wed, 09 Nov 2005 17:28:50 GMT

Sally -
James nailed it with newer.
James -
This is a great tip! Wasn't aware of it, but will definitely use it.

Thanks.

And Sally, please no points on this reply, just wanted to acknowledge anothers great contribution.

Regards,

dl

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 09:10:30 GMT

Thanks for all the responses....I think James hit it with "newer". I am going to try that.

S.

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 09:23:44 GMT

hmmmmmm....I sent my previous reponse too soon. I am wondering, would the files have to have the same name with "newer"? The problem is that the name always changes...the one consistent thing is that it is always a tar file. However, I can't say "newer *.tar". What I want to accomplish is to check the ftp server throughout the day for the new tar file. If the tar file that exists is what I already have in my directory, then done. If it is a new file, that I don't have...download it.

Thanks,

Re: Automate FTP

Rick Garland — Thu, 10 Nov 2005 09:29:26 GMT

Use mget (instead of get) and specify 'newer *.tar'

The mget accepts wildcards.

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 09:36:33 GMT

I tried that, and this was what came back:

ftp> mget newer *.tar
newer: The system cannot find the file specified.
mget dat-4624.tar? y
200 PORT command successful.
150 Opening ASCII mode data connection for dat-4624.tar(7710720 bytes).

Then it starts "getting" the file. I must not be using the correct syntax or something. Shouldn't it determine that there is only one tar file there, so that would be the one....and it is not newer than what I have and therefore not download it?

Re: Automate FTP

Rick Garland — Thu, 10 Nov 2005 09:46:38 GMT

What happens when you turn off the prompt?

prompt off
mget newer *.tar
...

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 09:53:57 GMT

It starts getting the file. However, I already have that one, so I don't want it to get it.

prompt off
Interactive mode off.
ftp> mget newer *.tar
newer: The system cannot find the file specified.
local: dat-4624.tar remote: dat-4624.tar
200 PORT command successful.
150 Opening ASCII mode data connection for dat-4624.tar(7710720 bytes).

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 10:01:18 GMT

Is there an equivalent to using grep within an ftp session? Like, if I could list the files and grep on the tar files, that would leave that one...then awk everything but the name of the file to a variable...and mget newer $var ? or is that asking too much of poor ftp?

Re: Automate FTP

A. Clay Stephenson — Thu, 10 Nov 2005 10:01:24 GMT

The most important glaring problem with these other approaches is that there is zero error checking. If you use the method I suggested and check the exit status of the Perl script itself then if it's zero then all is well -- meaning the complete file got there intact. The good news (maybe) is that you really don't have to know any Perl at all. All of the logic is still in the shell script that calls the Perl script. You simply have to feed it arguments and check the exit status.

Re: Automate FTP

Rick Garland — Thu, 10 Nov 2005 10:03:50 GMT

If you use the -g option (noglobbing) instead of the newer, will that still download the file?

Re: Automate FTP

Coolmar — Thu, 10 Nov 2005 10:06:57 GMT

You mean start the seesion with -g then mget *tar?

ftp -g
mget *.tar

BTW, what does "globbing mean"