topic Re: Getting files via sftp in Operating System - Linux

Getting files via sftp

Gyankr — Fri, 07 Dec 2007 15:10:48 GMT

Hi,
I have a remote server which publishes flat files everyday.I want to write a shell script which sftp's to the remote server and gets the "latest file". I was thinking how to write the script which first logs on to the server,changes to the directory where the files are and then gets them to the local directory. I am not an expert in sftp,any thoughts on this.

Regards,
Gyan

Re: Getting files via sftp

A. Clay Stephenson — Fri, 07 Dec 2007 15:48:30 GMT

If I were doing this, I would use Perl's Net::SFTP module. The module takes less than 1 minute to install and has all the primitives that you need. Moreover, it is trivially easy to check the status of each operation so that error checking goes from being a tedious exercise to a very simple operation.

http://search.cpan.org/~dbrobins/Net-SFTP-0.10/lib/Net/SFTP.pm

Re: Getting files via sftp

Autocross.US — Fri, 07 Dec 2007 16:07:34 GMT

If you setup ssh key exchange on both hosts you wouldn't need to script the login part.

You could get the file like this (on local host):
cd /local/dir/to/put/file
sftp USER@host2:/path/to/remote/file .

The challenge is to determine which file is the latest file. Can you provide details on this file? Is it time stamped or does it overwrite an existing file?

Re: Getting files via sftp

Gyankr — Tue, 11 Dec 2007 07:31:46 GMT

the filenames have the time stamp in it like CMO_LANG_STD_2007-11-23.dat and yes we are using public/private keys.I thought of sftp -b option but i cannot write shell scripts in the batch file :(

Re: Getting files via sftp

Autocross.US — Wed, 12 Dec 2007 21:13:36 GMT

Assuming you have keys working from host1 to host2, you could try something like this to determine the most recent file on host2 from host1:

FILES=`ssh host2 find /directory/with/files/ -type f -mtime -1 -name "CMO_LANG_STD_\*"`

cd /local/dir/to/put/file
for i in $FILES ; do
sftp USER@host2:$i .
done

An easier solution would be to sftp the file from host2 to host1. Then you only need 1 sftp command. You could use the find command to determine the most recent file.

Re: Getting files via sftp

Ben Dehner — Mon, 17 Dec 2007 23:26:42 GMT

At the risk of starting a flamewar, I'm not a big fan of any Perl CPAN modules. I'm not familiar with this one, but I've noticed in that past a recursion problem compiling Perl modules, such that module A needs module B needs module C ...

The biggest problem you face is determing what is the "latest" file. Instead of using sftp, it might work better with scp with a wildcard:

scp -p remotehost:/remote/dir/CMO.* .

once the file has downloaded, chmod u-w so that the next invocation of scp won't over-write and re-fetch the same file. The "-p" option will preserve the access times so you can see what the latest version is.

Re: Getting files via sftp

James R. Ferguson — Mon, 17 Dec 2007 23:37:48 GMT

Hi:

> Ben: At the risk of starting a flamewar, I'm not a big fan of any Perl CPAN modules. I'm not familiar with this one, but I've noticed in that past a recursion problem compiling Perl modules, such that module A needs module B needs module C ...

If you use CPAN to fetch modules you can configure the process to either ask or automatically follow any dependencies. There is no need to manually "chase" anything.

# perl -MCPAN -e shell

Regards!

...JRF...

Re: Getting files via sftp

Gyankr — Wed, 19 Dec 2007 13:23:44 GMT

Thanks for your replies