script to move files

[NDO]

Hi all

I have a script that move files by month, but somehow it is not working, and returns the following error:
root@nrtrde2 # ./move.sh

./move.sh: line 4: /usr/bin/scp: Arg list too long


The script is :
find /global/xnode/mcel/tap_out -type f -exec ls -lrt {} \; |
awk '{if ($6 == "Apr" && $8 != 2010) print $9}' > file_list

scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Can someone help

F.R.

[James R. Ferguson]

Hi:

The culprit line is this:

# scp -p $(< file_list) ...

There are too many arguments (filenames) and/or the total length is too long.

You may be forced to divide-and-conquer by either running several passes to collect files or by adding code to divide the whole file list into a series (maybe 3-10) of smaller subfiles that you then pass along to 'scp' from a 'for' loop.

You can also gain performance by changing the ':' terminator of your 'find's '-exec' to a '+'. This causes multiple arguments to be collected and passed to the pipeline, thereby dramatically reducing the number of 'awk' processes that are spawned. Your users may appreciate this.

Regards!

...JRF...

[NDO]

Hi

The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...



F.R.

[James R. Ferguson]

Hi (again):

> The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...

That's not the issue. The issue is that the *results* of the 'find' are too vast to form an argument list on the 'scp' command line.

Divide-and-conquer as I said above.

You could also break your 'find' into chunks by doing something like:

# touch -mt 201104010000 /tmp/ref1
# touch -mt 201104152359 /tmp/ref2
# find find /global/xnode/mcel/tap_out -type f \( -newer /tmp/ref1 -a ! -newer /tmp/ref2 \) >> file_list
# scp ...

Wrap this logic in a loop that varies the timestamps of the reference files created and walk through the timeframe you need.

This also eliminates the majority of the forks you original code does!

Regards!

...JRF...

[NDO]

Hi

I have followed your advise and now I am getting issues on the "scp"

I am having a syntax error.
can you share with me your knowledge, please.


F.R.

[James R. Ferguson]

Hi:

> I have followed your advise and now I am getting issues on the "scp" I am having a syntax error.

Post your script. WIthout it and the actual error I can only guess ...

Regards!

...JRF...

[James R. Ferguson]

HI (again):

I meant to have written this block :

...
find /global/xnode/mcel/tap_out -type f \( -newer /tmp/ref1 -a ! -newer /tmp/ref2 \) > file_list
scp -p $(< file_list) ...

That is, a redirection that overwrites the 'file_list' each time though the loop, not one that appends to the contents each time :-(

Regards!

...JRF...

[NDO]

Hi

Sorry for the late reply:

when I run:
scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

I got the following error:
syntax error: `(' unexpected

Regards

Fernando

[Dennis Handly]

>I got the following error: syntax error: `(' unexpected

Nothing obvious. Perhaps you have a file with a "("? Try:
echo scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Though if you aren't using a real shell, like the scummy C shell, you may get errors like this on the $() syntax. Use `` instead.

[NDO]

Hi


I did try that, see the o/p:

root@nrtrde2 # echo scp -p $(< file_list)
syntax error: `(' unexpected
root@nrtrde2 # echo $SHELL
/sbin/sh


And there is no special characters in file_list

Regards

Fernando