topic Re: download scripting in Operating System - Linux

download scripting

Jerome Henry — Tue, 25 Nov 2003 11:09:43 GMT

Hi everybody,

I'm wishing to write a script that will :
1. open an ftp server myserver.com
2. Move to a defined directory /somedir
3. Check if files there are different or more recents than the files I already downloaded in localhost/mydirectory from this server.
4. Download the new or more recent files, ignoring files older than, say 120 days.
5. Check in localhost/mydirectory if files here came to the 120 days dead end (files downloaded before, of course).
6. Delete these old files.

Any proposition ?

5 points for any new part,
2 points for an addition or improvement,
and of course a rabbit for the full script.

Thanks.

J

Re: download scripting

John Poff — Tue, 25 Nov 2003 14:41:29 GMT

Hi J,

I've played with something like this before. I'd suggest writing the script in Perl and using the Net::FTP module. It will handle all your FTP connection and file commands and it will make life much easier.

Take a look at it here:

http://search.cpan.org/~gbarr/libnet-1.17/Net/FTP.pm

JP

Re: download scripting

Stuart Browne — Wed, 26 Nov 2003 01:35:51 GMT

There's a pre-written script around called 'mirror'. It does pretty much everything you've asked.

Failing that, ncftpget server:/path/, and a find would do most of that for you ;)

Re: download scripting

Vitaly Karasik_1 — Wed, 26 Nov 2003 02:41:30 GMT

rsync may help you too

Re: download scripting

Steven E. Protter — Wed, 26 Nov 2003 03:00:02 GMT

http://www.hpconsulting.com/Cleantmp

That script clears out tmp files over seven days old. There is some interactive junk to clear out but it works great in batch mode, change the target directory for the find and the number days to deal with your 120 day requirement

I would suggest running this script and modifying it to move eligible files to a staging directory instead of the rm command it currently uses.

Here is the ftp part:

ftp -v -n <open myserver.com
user username password
cd /somedir
lcd localhost/mydirectory
put filename
EOF

A while loop can be but around the ftp command running it multiple times based on the list producted in the first script.

The Script at the top can have a second copy run to delete the 120 day old targets. The government might not let you clone humans, but this script likes being cloned. That covers 5 and 6.

I just drove to STL from Chicago and my daughter won't sleep. Given a couple of hours, I can produce you a working, tested script. In case you are in a hurry however, my pieces will work.

I would suggest the possiblity of using openssh to skip the ftp part and make the file transfer somewhat secure.

Let me know if you want a finished product. When I deliver depends on factors beyond my control at this time.

SEP

Re: download scripting

Huc_1 — Wed, 26 Nov 2003 06:49:17 GMT

Hey your having all the fun here !

I have been job hunting (former company now for sale), forgot how much time this exercice required, it as keep me busy theses last few day's !, Jerome this is a good exercice for RHCE exams I will have a go at it this evening if someone else did not beat me to it .. specialy SEP ...

but question ? you want only ftp and bash ? could we use sftp or rsync ?

J-P

Re: download scripting

Cristian Draghici — Wed, 26 Nov 2003 10:25:01 GMT

wget -N ftp://username:password@some.server.com/dirtosync

will do the trick except:
- it does not ignore files on the FTP server older than 120 days
- it does not delete local files older than 120 days

tmpwatch is the easy way to delete old files
- add in cron.daily (or monthly)
/usr/sbin/tmpwatch 240 /dir/to/prune

I don't think you can retrieve files based on date without writing a FTP application.

If you really need to get based on date I think your best bet would be to use the ncftpspooler, ncftpbatch, ncftpget suite.

Cheers
Cristi

Re: download scripting

Patrick Van Humbeeck — Wed, 26 Nov 2003 10:25:55 GMT

why not simply use wget ? It can do FTP and has an option to only download files if they're newer than the local copies, and since it's command line you can easily include it in scripts.

the deletion is pretty straight forward, run something like
find /path -mtime +120 -exec rm {} \;

Re: download scripting

Steven E. Protter — Wed, 26 Nov 2003 11:13:31 GMT

hpuxconsulting.com is whacked again

http://www.hpux.ws/Cleantmp

Is up so try that one.

I'm solo on Parent patrol so it will be a while before I can try that script.

SEP

Re: download scripting

Jerome Henry — Thu, 27 Nov 2003 16:42:51 GMT

I'm back !

Benn traveling to attend Computer security fair in Paris. Fun to see that many vendors know their product so well and security so bad ! Is it the same in USA ?

Back to the point. Thanks everyone for working on my prob.

John, thanks for the idea. But I must confess I'm a perl dummy. I know how to use a program, but am very weak at wrtting any change on it... unless it's done for me :]] But why not, with a few help...

Stuart, I've heard about mirror program, but can't find it back : do you have any pointer ?

Vitaly, thanks for rsync idea, but the trouble is that I do not want to re-download the full directory, as I'm throwing away old programs, which the original directory doesn't...

Steven, Thanks for the link, I got it from hpux.ws. I'll try it tommorrow (late here...) and tell you who it turns to bash (it's ksh, but doesn't seem to be too sepcific :] )

Christian, yes, that's the issue, not redownloading what I will delete anyway...

Patrick, you may have it ! Looking at the scripts you suggest seems to be a workable solution : once again, 'll try it tommorrow !

Jean Pierre, as an RHCE, why not suggest something on rsync and find, even incomplete ?

I'll assign points too tommorrow : any last suggestion ?

Thanks a lot anyway !

J

Re: download scripting

Stuart Browne — Thu, 27 Nov 2003 18:16:20 GMT

RSync isn't nice enough to deal with files of any given age. A URL for the mirror perl script though:

ftp://ftp.cs.columbia.edu/archives/perl/mirror/

It has the ability to delete files older than 'max_days' of age.

I'm not sure if it will also delete local files (I think it will).

It's old, circa 2000, but it was a fantastic tool then, and is a good one for the war-chest.

Re: download scripting

Huc_1 — Fri, 28 Nov 2003 03:59:18 GMT

Sorry for late reply but, I have been busy lately
What I had in mind is something like the following

I first would delete old from remote structure as your question made me think you had delete rights on remote directory !

then rsync remote to local

then clean old on local

Something like this

ssh JeromeHenry@myserver.com 'find /somedir -mtime +120 -exec rm {} \;'
rsync -avz -e ssh myserver.com/somedir /mydirectory
find /mydirectory -mtime +120 -exec rm {} \;

a very busy, but having some positive replies

J-P

Re: download scripting

Huc_1 — Fri, 28 Nov 2003 10:41:22 GMT

Please remplace
rsync -avz -e ssh myserver.com/somedir /mydirectory

with

rsync -avz -e ssh JeromeHenry@myserver.com:/somedir /mydirectory

forgot the ":"

I ssh is setup right pn source and target it does not ask for password.

J-P (0 pts for this mea-culpa and add on)

Re: download scripting

Olivier Drouin — Fri, 28 Nov 2003 12:44:49 GMT

here, some ftp functions ( 1 for ssh/scp, the other for standard ftp ) written in bash.

just source the file within another script and the functions will be available in your script.

Re: download scripting

Olivier Drouin — Fri, 28 Nov 2003 12:56:57 GMT

Like Huc said you can add the public key of a user on a server to your ~/.ssh/known_hosts on another server and ssh will not bother you with passwd.

Re: download scripting

Jerome Henry — Sat, 29 Nov 2003 03:13:11 GMT

Thanks everyone !
Steven and Stuart's tips are 2 100% workable solution, ready to use.
Jean Pierre's is also workable, except that I do not have write rights on remote server... so should it have to be turned slightly differently to take this into consideration.

Thanks again for all this work ! I consider this thread as solved.

I'll have another question in 10/12 days :]]

J

Re: download scripting

Jerome Henry — Sat, 29 Nov 2003 07:29:51 GMT

Hi,

Slight modification in find script :
find /path -ctime +120 -exec rm -f {} \;
I prefer -ctime to search for creation date instead of modification, then use -f not to be asked at each line if I'm sure to remove this file, and space is required between {} and ;

Thanks again !

J