script to copy file from several directories

[NDO]

Hi All

I need to scp files from several subdirecories within the same directory to another server.

Pls can you help?

F.R.

[James R. Ferguson]

Hi:

What are you stuck with?

Setup public ssh keys between the servers in question. Then write whatever 'scp' command you need! For example:

# scp -p mydir/mysubdir1/myfile1 thathost:/itsdir/itsfile

Search or Google ssh and public keys if you don't know how to set that up. The manpages for 'scp' would also help you.

Regards!

...JRF...

[Mel Burslan]

scp mysubdir1/myfile1 mysubdir2/myfile2 mysubdir3/myfile3 remote_host:/target_directory

it is that simple

if the destination for all these files are different too, that that's another story and each different destination will need a separate scp command.

Hope this helps

[NDO]

Hi

The problem is that I need to scp all february files from 318 directories to just one destination. But the source are 318 subdirectories within one directory . I know how to select the february files. what I dont want is to create 318 scripts and run them.

F.R.

[Mel Burslan]

February files ??? As you can imagine, we, on the other side of the internet, do not know what this means. But my assumption is, you have a directory hierarchy as such:

/logs/2011/Jan
/logs/2011/Feb
/logs/2011/Mar
/logs/2011/Apr

and each of these have 300+ subdirectories under each of them. Right ? If so, let's take Feb subdirectory as an example since you mentioned it.

cd /logs/2011/Feb
scp -rp * remote_server:/var/logs/Feb


make sure on the remote server, directory /var/logs/Feb exists or if you want scp to create it, use this syntax:

cd /logs/2011
scp -rp Feb remote_server:/var/logs

again make sure the /var/log target directory exists on the remote server.

switch -r means copy subdirectories recursively and -p means preserve modification times, access times, and modes from the original file.

Hope this helps. More details you provide instead of rounding up your question, better answers you get around here. Just remember that next time you are about to ask a question. We are not mind readers unfortunately

[Steven Schweda]

> [...] We are not mind readers unfortunately

It might be fortunate. Imagine what we might
see if we could. Gahhh.

> But my assumption is, [...]

We should not need to assume anything.

> [...] another server.

For example, we should not need to assume
what "another server" is, or its OS.

> I need to scp files [...]

I doubt it.

> [...] 318 directories [...]

Is there some reason not to use, say, "tar"
to create one archive containing all these
files, and then copy _that_ to "another
server"?

Perhaps you should start by trying to decide
(and describe) exactly _what_ it is that you
need to do, and _then_ worry about exactly
_how_ you "need" to do it.

Or, you could just continue to waste
everyone's time the same way.

[Dennis Handly]

>I know how to select the february files. what I don't want is to create 318 scripts and run them.

You don't need 318 scripts but you may need 318 lines in your script.

Or if the target is in one directory, you can list 318 source files, like Mel mentioned:
scp \
file1 \
[...] \
host2:target-directory

If you have a script that can select those files, you can replace the above by:
scp $(selection-script.sh) host2:target-directory

[NDO]

Hi

let me explain again:
lets say I have directory Europe, inside this directory I have 318 subdirectories all under Europe. Now under this 318 subdirectories are thousand of files, all created in differente months, days etc. So what I want is to scp only the files created in february of this 318 subdirectories to a single destination on another server. So I have multiple sources (318) but only one target.


F.R

[NDO]

hi

the script I use for one directory is as follows:

for JOB in `ls -lrt | awk '{if ($6 == "Feb" && $8 != 2010)print $9}'`
do
scp -p /global/xnode/mcel/tap_out//$JOB root@xx.xxx.xx.xx:/dcs/data02_loc/PROD/INPUT/TAPOUT/

F.R.

[Viktor Balogh]

Hi,

First, I would create two marker files. file1 should have a timestamp of february 1st, file2 february 28.

# touch -t 201102010000 file1
# touch -t 201102282359 file2

With these two, the following find command will result in all the files you need to copy:

# find /Europe -newer file1 -a ! -newer file2 -print

With this, you can generate a file list, tar it and scp it. I'm not sure if you want to copy these with full path or not, but you can achieve both with tar.

you can use the exec option of find:

# find /Europe -newer file1 -a ! -newer file2 -exec tar cvf {} \;

or you can use pax if you prefer it:

# find /Europe -newer file1 -a ! -newer file2 | pax -v -w -f /tmp/myarchive

Regards,
Viktor

[Dennis Handly · Accepted solution]

>the script I use for one directory is as follows:

Change to:
find /global/xnode/mcel/tap_out -type f -exec ll -rt + |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list

scp -p $(< file_list) root@xx.xxx.xx.xx:/dcs/data02_loc/PROD/INPUT/TAPOUT/

You don't have to produce file_list if you want to go crazy and put it all on one line but you may want to keep it around with the two steps.

You can also add "-mtime -$(date +%j)" to find(1) to limit the search to just this year.

[Dennis Handly]

>Viktor: but you can achieve both with tar.

Don't even think of using find(1) with tar.
Unless the list is "small":
tar -cvf foo.tar $(find /Europe -newer file1 -a ! -newer file2)

# find /Europe -newer file1 -a ! -newer file2 -exec tar cvf {} \;

This puts the file in itself? :-(
Or overwrite the same tarfile you forgot.

>you can use pax if you prefer it:
# find /Europe -newer file1 -a ! -newer file2 | pax -v -w -f /tmp/myarchive

Yes.

[NDO]

Hi

>>Change to:
find /global/xnode/mcel/tap_out -type f -exec ll -rt + |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list



the output is:
find: incomplete statement


F.R.

[Viktor Balogh]

I see that you want the hard way (ll & awk). try this:

find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+ |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list


in short: you need a {} and a + or ; at the end of -exec.

[NDO]

Hi

the output its empty file (file_list)

F.R.

[Viktor Balogh]

>the output its empty file (file_list)

first, try without awk...

[NDO]

Hi

>>first, try without awk...


I dont understand.


F.R.

[Viktor Balogh]

at first try only this:

# find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+

does it give any o/p?

[Dennis Handly]

>the output is: find: incomplete statement

This works fine on HP-UX's find(1), using a real shell. What are you using?

>Viktor: in short: you need a {} and a + or ; at the end of -exec.

On HP-UX you don't need that "{}" before the "+" but I probably shouldn't have left it out in my example.

>at first try only this. does it give any o/p?

Right. Is /global/xnode/mcel/tap_out the correct path to "Europe"? Does ll(1) produce any output?

[NDO]

Hi

Sorry, I was away for some days:
yes
" find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+
" does give me an output:

-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/BENSP/CDMOZ01BENSP02186
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/NERCT/CDMOZ01NERCT01288
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/ARGCM/CDMOZ01ARGCM01811
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/ALBAM/CDMOZ01ALBAM01766


F.R.

[James R. Ferguson]

Hi (again):

> yes
" find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+
" does give me an output:

OK, that's nice. Now, does this help you to find a solution to your original question?

Regards!

...JRF...

[NDO]

Hi

James: Yes, I am now able to copy all files from the 318 directories without having to write 318 scripts. I will assign some points now.


F.R.

[NDO]

bye

[Viktor Balogh]

my first proposal with the marker files would give you the same files,

# find /Europe -newer file1 -a ! -newer file2 -print

and you wouldn't need those ll and awk commands. ll takes more time as it reads several properties from the inodes, while the find command would only touch the timestamp file of each inode. So I think it would yield in the same results but much faster.

Anyway, thank you for the points and I'm glad it solved your problem.

[Viktor Balogh]

timestamp file --> timestamp field