remote file copy from multiple directories

[Olebile]

Distinguished Gentlemen,

I am experiencing a mental block on this one and I need your help.

I want to schedule a job on cron that will copy files from multiple directories daily from a remote system. With each day I want to copy only new files from the remote directories. The remote host where the files are follows the following parttern

/BTC/Dir1/tertiary/*
/BTC/Dir2/tretiary/*
/BTC/Dir3/tertiary/*

and so on.

Where do I start such that I can create a loop to copy daily from the remote system to local using cron?

Thx

[Jarle Bjorgeengen]

I'd use rsync for this operation, either through ssh (exchanging ssh-keys for passowrdless access) or setting up rsync server on the directories to be copied from.

- JarleB

[Peter Godron]

Hi,
I would start by looking at this earlier thread:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1109365

If you insists on ftp:
You would ftp to the machine and then use the cd command to change directories.

Please also read:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33 on how to reward any useful answers given to your questions.

So far you have rewarded 7 of 113 answers !

[Dennis Handly]

Are these remote directories mounted with NFS so you can look at the timestamps and use find -newer?

[Olebile]

I have never done this before except easy "For" loops on the command line.

Please kindly guide me with sample text that I can customise?

Regards

[Olebile]

Dennis,

The 2 systems are standalones on the same network. I could NFS mount the remote filesystem onto the local host. Then I will need to copy the files to the local machine so that I can manipulate them with changing the source.

I know how to NFS mount, how do I then move forward so that I can have a cron that copies the files to a local directory?

[Dennis Handly]

>Please kindly guide me with sample text that I can customise?

We need to know how you are going to access the files. If NFS, it is easy. If pushing by ftp, still easy. If pulling over ftp, and you can't remove the files you copied, you are hosed and must turn to advanced AI technology. You must copy all files and then remove the dups. Or you must parse the name/size/timestamp and see if you already have done it. And if not, you must ftp-get the file.

As JarleB says, if you have to use ftp, you should look at rsync instead.

[Olebile]

I would prefer to use NFS in this case because this will be a daily task that I would like to schedule on cron in the local host.

[Dennis Handly]

>I would prefer to use NFS.

This is good. You should look at my solution in Peter's link above. Then replace the "ftp" plugin by a cp.

[Olebile]

I have seen the syntax for FTP, I can't seem to figure-out how I will send the copy thru "cp" into a loop so that I can copy from the various directories:

/BTC/Dir1/tertiary/*
/BTC/Dir2/tretiary/*
/BTC/Dir3/tertiary/*
/Btc/Dir4/tertiary/*

Please kindly assist with sample syntax?

[Peter Nikitka]

Hi,

will there be different local directories
Dir1/
Dir2/
...
as well or differs the local from the remote structure?

If it is equivalent, adapt+try the following code fragment. It will coy file not already local - you can change this condition to your needs:

nfsdir=/nfsmount/remhost/BTC
locdir=/archive/btc
inter=tertiary

for dir in Dir1 Dir2 ...
do
[ -d $locdir/$dir/$inter ] || mkdir $locdir/$dir/$inter
for f in $nfsdir/$dir/$inter/*
do
[ -f $locdir/$dir/$inter/${f##*/} ] && continue
cp -p $f $locdir/$dir/$inter
done
done

mfG Peter

[Olebile]

The final bits now, when the copy happens it should only copy files from last.

Where do i stick somethin like this:

for i in `find . -mtime 1 -print |xargs ls -l | awk '{print $9}''


Regards

[Olebile]

Peter,

Correct the logic here?

We are going to mount the remote filesystem onto the local host thru NFS.

We do a copy from the NFS filesystem sub-directories to a seperate but 1 directory on the local host.

The copy should only copy files from last nite. Example if today is the 20th the copy should get files for the 19th.

How do we place the above into a script?

[Olebile]

Gentlemen,

The "for" loop below is what I tried to use but it is limited to

/BTC/DataMediation/L1/tertiary/
/BTC/DataMediation/L2/tertiary/
/BTC/DataMediation/L3/tertiary/
/BTC/DataMediation/L4/tertiary/

It copies from the various L1,L2,L3 directories files that are a day old. To another filesystem /backup/Transfer.


I wanted to expand the logic above into a loop such that I can copy from other subdirectories in the same manner. Eg

/BTC/DataMediation/G1/tertiary/
/BTC/DataMediation/G2/tertiary/
/BTC/DataMediation/G3/tertiary/

To do this I need a to create a loop, that where I got stuck?

Please Help

SAMPLE SYNTAX:

For i in

'find /BTC_DataMediation/L*/tertiary/ -mtime 1 -print|xargs ls -l|awk '{print $9}''

do

cp -p $i /backup/Transfer/

done

[Peter Nikitka · Accepted solution]

Hi Olebile,

there where many threads in this forum dealing with timebased selection of files - nevertheless:
I suggest to take the newest local file as a reference for the decision to copy-or-not - though can can modify this to a solition using a file created by 'touch '.

If you copy all remote files, found in different directories, to a single directory, there is the possiblity, that the plain filenames are not unique and one file is overwriting the other. Your scenario may not have this inpact, however.

nfsdir=/nfsmount/remhost/BTC
locdir=/archive/btc
inter=tertiary

touch nowfile
newest_got=$(ls -t $locdir | sed 1q)

for dir in Dir1 Dir2 ...
do
for f in $nfsdir/$dir/$inter/*
do
if [ -n "$newest_got" ]
then
[ $f -ot $locdir/$newest_got ] && continue
fi
[ $f -nt nowfile ] && continue
cp -p $f $locdir
done
done
rm nowfile

mfG Peter

PS: Your profile states:
"I have assigned points to 7 of 117 responses to my questions."

You really should change that.

[Dennis Handly]

>for i in `find . -mtime 1 -print |xargs ls -l | awk '{print $9}''

Did you see my solution regarding a reference file?

If you want to use find, no need to do for, xargs or ls. You should do:
find /BTC_DataMediation/L*/tertiary/ -mtime 1 -print | while read file; do
    cp -p $file /backup/Transfer/
done

If you know you aren't finding "lots" of files, then you can use:
for file in $(find ... ); do
    cp -p $file /backup/Transfer/
done

[Olebile]

Peter,

I see the logic in your script, I'll test it and give the results. The files will not overwrite because they are all unique.

Meanwhile the other suggested solution are much appreciated but they do not give me a copying solution based on time.

The reason why it's time based is that I don't want to copy everything from the remote directory that was already copied.

Thx

[Dennis Handly]

>The reason why it's time based is that I don't want to copy everything

My script in the following link can be changed to be based on time:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1109365

You just need to use what Peter has here:
[ $f -nt nowfile ]
to sort the result of find of the timestamps on the original machine.

[Olebile]

Peter,

Your script does work I think just one tweek will make it work 100%.

Please look at the files below at the source:

-rw-rw-r-- 1 root other 54192 Mar 21 01:17 L1.TTFILE00.1727.gz
-rw-rw-r-- 1 root other 51900 Mar 21 01:28 L1.TTFILE02.1725.gz
-rw-rw-r-- 1 root other 20139 Mar 21 03:13 L1.TTFILE00.1728.gz
-rw-rw-r-- 1 root other 17793 Mar 21 03:24 L1.TTFILE02.1726.gz
-rw-rw-r-- 1 root other 75776 Mar 21 05:11 L1.TTFILE00.1729
-rw-rw-r-- 1 root other 69632 Mar 21 05:23 L1.TTFILE02.1727
-rw-rw-r-- 1 root other 53248 Mar 21 07:11 L1.TTFILE00.1730
-rw-rw-r-- 1 root other 51200 Mar 21 07:23 L1.TTFILE02.1728
-rw-rw-r-- 1 root other 180224 Mar 21 09:12 L1.TTFILE00.1731
-rw-rw-r-- 1 root other 196608 Mar 21 09:23 L1.TTFILE02.1729
-rw-rw-r-- 1 root other 299008 Mar 21 2007 L1.TTFILE00.1732
-rw-rw-r-- 1 root other 323584 Mar 21 2007 L1.TTFILE02.1730
-rw-rw-r-- 1 root other 1464320 Mar 21 2007 L1.TTFILE00.1733
-rw-rw-r-- 1 root other 1536000 Mar 21 2007 L1.TTFILE02.1731

Wed Mar 21 09:59:36 SAST 2007

Script Output:

-rw-rw-r-- 1 root other 180224 Mar 21 09:12 /nfscdm/L1/tertiary/L1.TTFILE00.1731
-rw-rw-r-- 1 root other 196608 Mar 21 09:23 /nfscdm/L1/tertiary/L1.TTFILE02.1729
-rw-rw-r-- 1 root other 18432 Mar 21 08:30 /nfscdm/L2/tertiary/L2.TTFILE00.1576
-rw-rw-r-- 1 root other 20480 Mar 21 08:31 /nfscdm/L2/tertiary/L2.TTFILE02.1575

Looking at the above you will notice that the output files from the script have the right date but they are fewer than the files at the source. Therefore not all files are copied. I think the "touch nowfile" line in the script has something to do with this. Is it ok to assume that if the script ran at an ealiear time before 0900Hrs like I did, I will get all files?

[Peter Nikitka]

Hi,

the statement

touch nowfile

will set a mark at the start of the script, that no files modified AFTER the start will be considered as copy candidates.
So the possibility is lowered (but is not zero) that you copy incomplete files. These skipped files will be copied at the next run of the script.

But look carefully at the difference of this lines:
-rw-rw-r-- 1 root other 196608 Mar 21 09:23 L1.TTFILE02.1729
-rw-rw-r-- 1 root other 299008 Mar 21 2007 L1.TTFILE00.1732

It seems to me be, that the system clocks of your local host and the NFS server differ and so the comparing of timestamp seems incorrect:
The output format of the second line tells us a timestamp in the future (relativly seen on the local host)!

Keep in mind, that we create the stamp (on file nowfile) on the local filesystem, so no write access over NFS is necessary.

mfG Peter

PS: I really considered NOT to reply to your last question, since you do not honour the work of the answering forum members by giving points:
>>
assigned points to 7 of 120 responses
<<

[Olebile]

Peter,

I appreciate your help, thax very much. It may seem I don't give points but I once the solution is found. It is not always that you get the right solution from each response.

You can see in the dialogue that your solution is different all the other and it is the one that met all requirements.

Maybe I should also give points for partial solutions and efforts.

PS: The timestamp on both systems is only a few seconds apart but timezone is GMT on 1 and SAST on the other

Thanks Again