Getting files via sftp

[Gyankr]

Hi,
I have a remote server which publishes flat files everyday.I want to write a shell script which sftp's to the remote server and gets the "latest file". I was thinking how to write the script which first logs on to the server,changes to the directory where the files are and then gets them to the local directory. I am not an expert in sftp,any thoughts on this.

Regards,
Gyan

[A. Clay Stephenson]

If I were doing this, I would use Perl's Net::SFTP module. The module takes less than 1 minute to install and has all the primitives that you need. Moreover, it is trivially easy to check the status of each operation so that error checking goes from being a tedious exercise to a very simple operation.

http://search.cpan.org/~dbrobins/Net-SFTP-0.10/lib/Net/SFTP.pm

[Autocross.US]

If you setup ssh key exchange on both hosts you wouldn't need to script the login part.


You could get the file like this (on local host):
cd /local/dir/to/put/file
sftp USER@host2:/path/to/remote/file .

The challenge is to determine which file is the latest file. Can you provide details on this file? Is it time stamped or does it overwrite an existing file?

[Gyankr]

the filenames have the time stamp in it like CMO_LANG_STD_2007-11-23.dat and yes we are using public/private keys.I thought of sftp -b option but i cannot write shell scripts in the batch file :(

[Autocross.US]

Assuming you have keys working from host1 to host2, you could try something like this to determine the most recent file on host2 from host1:

FILES=`ssh host2 find /directory/with/files/ -type f -mtime -1 -name "CMO_LANG_STD_\*"`

cd /local/dir/to/put/file
for i in $FILES ; do
sftp USER@host2:$i .
done

An easier solution would be to sftp the file from host2 to host1. Then you only need 1 sftp command. You could use the find command to determine the most recent file.

[Ben Dehner]

At the risk of starting a flamewar, I'm not a big fan of any Perl CPAN modules. I'm not familiar with this one, but I've noticed in that past a recursion problem compiling Perl modules, such that module A needs module B needs module C ...

The biggest problem you face is determing what is the "latest" file. Instead of using sftp, it might work better with scp with a wildcard:

scp -p remotehost:/remote/dir/CMO.* .

once the file has downloaded, chmod u-w so that the next invocation of scp won't over-write and re-fetch the same file. The "-p" option will preserve the access times so you can see what the latest version is.

[James R. Ferguson]

Hi:

> Ben: At the risk of starting a flamewar, I'm not a big fan of any Perl CPAN modules. I'm not familiar with this one, but I've noticed in that past a recursion problem compiling Perl modules, such that module A needs module B needs module C ...

If you use CPAN to fetch modules you can configure the process to either ask or automatically follow any dependencies. There is no need to manually "chase" anything.

# perl -MCPAN -e shell

Regards!

...JRF...

[Gyankr]

Thanks for your replies