topic Re: script to copy file from several directories in Operating System - HP-UX

script to copy file from several directories

NDO — Tue, 05 Apr 2011 14:04:17 GMT

Hi All

I need to scp files from several subdirecories within the same directory to another server.

Pls can you help?

F.R.

Re: script to copy file from several directories

James R. Ferguson — Tue, 05 Apr 2011 14:37:51 GMT

Hi:

What are you stuck with?

Setup public ssh keys between the servers in question. Then write whatever 'scp' command you need! For example:

# scp -p mydir/mysubdir1/myfile1 thathost:/itsdir/itsfile

Search or Google ssh and public keys if you don't know how to set that up. The manpages for 'scp' would also help you.

Regards!

...JRF...

Re: script to copy file from several directories

Mel Burslan — Tue, 05 Apr 2011 16:25:24 GMT

scp mysubdir1/myfile1 mysubdir2/myfile2 mysubdir3/myfile3 remote_host:/target_directory

it is that simple

if the destination for all these files are different too, that that's another story and each different destination will need a separate scp command.

Hope this helps

Re: script to copy file from several directories

NDO — Tue, 05 Apr 2011 18:25:46 GMT

Hi

The problem is that I need to scp all february files from 318 directories to just one destination. But the source are 318 subdirectories within one directory . I know how to select the february files. what I dont want is to create 318 scripts and run them.

F.R.

Re: script to copy file from several directories

Mel Burslan — Tue, 05 Apr 2011 18:40:07 GMT

February files ??? As you can imagine, we, on the other side of the internet, do not know what this means. But my assumption is, you have a directory hierarchy as such:

/logs/2011/Jan
/logs/2011/Feb
/logs/2011/Mar
/logs/2011/Apr

and each of these have 300+ subdirectories under each of them. Right ? If so, let's take Feb subdirectory as an example since you mentioned it.

cd /logs/2011/Feb
scp -rp * remote_server:/var/logs/Feb

make sure on the remote server, directory /var/logs/Feb exists or if you want scp to create it, use this syntax:

cd /logs/2011
scp -rp Feb remote_server:/var/logs

again make sure the /var/log target directory exists on the remote server.

switch -r means copy subdirectories recursively and -p means preserve modification times, access times, and modes from the original file.

Hope this helps. More details you provide instead of rounding up your question, better answers you get around here. Just remember that next time you are about to ask a question. We are not mind readers unfortunately

Re: script to copy file from several directories

Steven Schweda — Tue, 05 Apr 2011 19:12:00 GMT

> [...] We are not mind readers unfortunately

It might be fortunate. Imagine what we might
see if we could. Gahhh.

> But my assumption is, [...]

We should not need to assume anything.

> [...] another server.

For example, we should not need to assume
what "another server" is, or its OS.

> I need to scp files [...]

I doubt it.

> [...] 318 directories [...]

Is there some reason not to use, say, "tar"
to create one archive containing all these
files, and then copy _that_ to "another
server"?

Perhaps you should start by trying to decide
(and describe) exactly _what_ it is that you
need to do, and _then_ worry about exactly
_how_ you "need" to do it.

Or, you could just continue to waste
everyone's time the same way.

Re: script to copy file from several directories

Dennis Handly — Wed, 06 Apr 2011 01:21:11 GMT

>I know how to select the february files. what I don't want is to create 318 scripts and run them.

You don't need 318 scripts but you may need 318 lines in your script.

Or if the target is in one directory, you can list 318 source files, like Mel mentioned:
scp \
file1 \
[...] \
host2:target-directory

If you have a script that can select those files, you can replace the above by:
scp $(selection-script.sh) host2:target-directory

Re: script to copy file from several directories

NDO — Wed, 06 Apr 2011 05:09:29 GMT

Hi

let me explain again:
lets say I have directory Europe, inside this directory I have 318 subdirectories all under Europe. Now under this 318 subdirectories are thousand of files, all created in differente months, days etc. So what I want is to scp only the files created in february of this 318 subdirectories to a single destination on another server. So I have multiple sources (318) but only one target.

F.R

Re: script to copy file from several directories

NDO — Wed, 06 Apr 2011 05:58:39 GMT

hi

the script I use for one directory is as follows:

for JOB in `ls -lrt | awk '{if ($6 == "Feb" && $8 != 2010)print $9}'`
do
scp -p /global/xnode/mcel/tap_out//$JOB root@xx.xxx.xx.xx:/dcs/data02_loc/PROD/INPUT/TAPOUT/

F.R.

Re: script to copy file from several directories

Viktor Balogh — Wed, 06 Apr 2011 10:21:37 GMT

Hi,

First, I would create two marker files. file1 should have a timestamp of february 1st, file2 february 28.

# touch -t 201102010000 file1
# touch -t 201102282359 file2

With these two, the following find command will result in all the files you need to copy:

# find /Europe -newer file1 -a ! -newer file2 -print

With this, you can generate a file list, tar it and scp it. I'm not sure if you want to copy these with full path or not, but you can achieve both with tar.

you can use the exec option of find:

# find /Europe -newer file1 -a ! -newer file2 -exec tar cvf {} \;

or you can use pax if you prefer it:

# find /Europe -newer file1 -a ! -newer file2 | pax -v -w -f /tmp/myarchive

Regards,
Viktor

Re: script to copy file from several directories

Dennis Handly — Wed, 06 Apr 2011 10:29:53 GMT

>the script I use for one directory is as follows:

Change to:
find /global/xnode/mcel/tap_out -type f -exec ll -rt + |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list

scp -p $(< file_list) root@xx.xxx.xx.xx:/dcs/data02_loc/PROD/INPUT/TAPOUT/

You don't have to produce file_list if you want to go crazy and put it all on one line but you may want to keep it around with the two steps.

You can also add "-mtime -$(date +%j)" to find(1) to limit the search to just this year.

Re: script to copy file from several directories

Dennis Handly — Wed, 06 Apr 2011 10:34:51 GMT

>Viktor: but you can achieve both with tar.

Don't even think of using find(1) with tar.
Unless the list is "small":
tar -cvf foo.tar $(find /Europe -newer file1 -a ! -newer file2)

# find /Europe -newer file1 -a ! -newer file2 -exec tar cvf {} \;

This puts the file in itself? :-(
Or overwrite the same tarfile you forgot.

>you can use pax if you prefer it:
# find /Europe -newer file1 -a ! -newer file2 | pax -v -w -f /tmp/myarchive

Yes.

Re: script to copy file from several directories

NDO — Wed, 06 Apr 2011 11:01:34 GMT

Hi

>>Change to:
find /global/xnode/mcel/tap_out -type f -exec ll -rt + |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list

the output is:
find: incomplete statement

F.R.

Re: script to copy file from several directories

Viktor Balogh — Wed, 06 Apr 2011 11:06:07 GMT

I see that you want the hard way (ll & awk). try this:

find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+ |
awk '{if ($6 == "Feb" && $8 != 2010) print $9}' > file_list

in short: you need a {} and a + or ; at the end of -exec.

Re: script to copy file from several directories

NDO — Wed, 06 Apr 2011 11:40:36 GMT

Hi

the output its empty file (file_list)

F.R.

Re: script to copy file from several directories

Viktor Balogh — Wed, 06 Apr 2011 11:43:38 GMT

>the output its empty file (file_list)

first, try without awk...

Re: script to copy file from several directories

NDO — Wed, 06 Apr 2011 11:52:37 GMT

Hi

>>first, try without awk...

I dont understand.

F.R.

Re: script to copy file from several directories

Viktor Balogh — Wed, 06 Apr 2011 18:34:45 GMT

at first try only this:

# find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+

does it give any o/p?

Re: script to copy file from several directories

Dennis Handly — Thu, 07 Apr 2011 04:20:22 GMT

>the output is: find: incomplete statement

This works fine on HP-UX's find(1), using a real shell. What are you using?

>Viktor: in short: you need a {} and a + or ; at the end of -exec.

On HP-UX you don't need that "{}" before the "+" but I probably shouldn't have left it out in my example.

>at first try only this. does it give any o/p?

Right. Is /global/xnode/mcel/tap_out the correct path to "Europe"? Does ll(1) produce any output?

Re: script to copy file from several directories

NDO — Mon, 11 Apr 2011 13:54:38 GMT

Hi

Sorry, I was away for some days:
yes
" find /global/xnode/mcel/tap_out -type f -exec ll -rt {} \+
" does give me an output:

-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/BENSP/CDMOZ01BENSP02186
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/NERCT/CDMOZ01NERCT01288
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/ARGCM/CDMOZ01ARGCM01811
-rw-r--r-- 1 mcel other 123 Apr 11 08:55 /global/xnode/mcel/tap_out/ALBAM/CDMOZ01ALBAM01766

F.R.