script to move files

[NDO]

Hi all

I have a script that move files by month, but somehow it is not working, and returns the following error:
root@nrtrde2 # ./move.sh

./move.sh: line 4: /usr/bin/scp: Arg list too long


The script is :
find /global/xnode/mcel/tap_out -type f -exec ls -lrt {} \; |
awk '{if ($6 == "Apr" && $8 != 2010) print $9}' > file_list

scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Can someone help

F.R.

[James R. Ferguson]

Hi:

The culprit line is this:

# scp -p $(< file_list) ...

There are too many arguments (filenames) and/or the total length is too long.

You may be forced to divide-and-conquer by either running several passes to collect files or by adding code to divide the whole file list into a series (maybe 3-10) of smaller subfiles that you then pass along to 'scp' from a 'for' loop.

You can also gain performance by changing the ':' terminator of your 'find's '-exec' to a '+'. This causes multiple arguments to be collected and passed to the pipeline, thereby dramatically reducing the number of 'awk' processes that are spawned. Your users may appreciate this.

Regards!

...JRF...

[NDO]

Hi

The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...



F.R.

[James R. Ferguson]

Hi (again):

> The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...

That's not the issue. The issue is that the *results* of the 'find' are too vast to form an argument list on the 'scp' command line.

Divide-and-conquer as I said above.

You could also break your 'find' into chunks by doing something like:

# touch -mt 201104010000 /tmp/ref1
# touch -mt 201104152359 /tmp/ref2
# find find /global/xnode/mcel/tap_out -type f \( -newer /tmp/ref1 -a ! -newer /tmp/ref2 \) >> file_list
# scp ...

Wrap this logic in a loop that varies the timestamps of the reference files created and walk through the timeframe you need.

This also eliminates the majority of the forks you original code does!

Regards!

...JRF...

[NDO]

Hi

I have followed your advise and now I am getting issues on the "scp"

I am having a syntax error.
can you share with me your knowledge, please.


F.R.

[James R. Ferguson]

Hi:

> I have followed your advise and now I am getting issues on the "scp" I am having a syntax error.

Post your script. WIthout it and the actual error I can only guess ...

Regards!

...JRF...

[James R. Ferguson]

HI (again):

I meant to have written this block :

...
find /global/xnode/mcel/tap_out -type f \( -newer /tmp/ref1 -a ! -newer /tmp/ref2 \) > file_list
scp -p $(< file_list) ...

That is, a redirection that overwrites the 'file_list' each time though the loop, not one that appends to the contents each time :-(

Regards!

...JRF...

[NDO]

Hi

Sorry for the late reply:

when I run:
scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

I got the following error:
syntax error: `(' unexpected

Regards

Fernando

[Dennis Handly]

>I got the following error: syntax error: `(' unexpected

Nothing obvious. Perhaps you have a file with a "("? Try:
echo scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Though if you aren't using a real shell, like the scummy C shell, you may get errors like this on the $() syntax. Use `` instead.

[NDO]

Hi


I did try that, see the o/p:

root@nrtrde2 # echo scp -p $(< file_list)
syntax error: `(' unexpected
root@nrtrde2 # echo $SHELL
/sbin/sh


And there is no special characters in file_list

Regards

Fernando

[Dennis Handly]

# echo scp -p $(< file_list)
syntax error: `(' unexpected

Then use quotes: echo "scp -p $(< file_list)"

># echo $SHELL

This only works if you don't invoke a different shell.
Try: ps -fp $$

[NDO]

Sorry Dennis!

I am lost:

root@nrtrde2 # ls file_list
file_list
root@nrtrde2 # echo "scp -p $(< file_list)"
scp -p $(< file_list)
root@nrtrde2 # ps -fp $$
UID PID PPID C STIME TTY TIME CMD
root 23099 23097 0 12:22:01 pts/1 0:00 -sh
root@nrtrde2 #


the echo just yields that


regards

F.R.

[Dennis Handly]

What HP-UX version are you on?
After you type escape in your real shell, what version does control-V show?

When I do that echo:
$ echo "scp -p $(< file_list)"
sh: file_list: Cannot find or open the file.

(Because I don't have file.)

About the only way that echo can produce that output is if the file "file_list" contains the string:
$(< file_list)

[NDO]

Hi James & Dennis


Its now working fine, what I did was to use as a second marker as file (/tmp/ref3) file with a much shorter period of time 201104052359.

Files are now being sent to the other server, but I will have create several chunks of markers in order to send all the April files.

Thank you for your help


regards

Fernando

[James R. Ferguson]

Hi (again) Fernando:

> Thank you for your help

If you are satisfied with the help that you received, please read the following about assigning points:

http://h30499.www3.hp.com/t5/help/faqpage/faq-category-id/kudos#kudos

Regards!

...JRF..

[NDO]

Yes James, I always assign points to everybody, but just one more query.

I did run the those comands on the prompt. If I put those on a script will it work?


regards


Fernando

[James R. Ferguson · Accepted solution]

Hi (again) Fernando:

> I did run the those comands on the prompt. If I put those on a script will it work?

Yes. However, if you plan to run your script from a 'crontask' remember that 'cron' establishes a minimal environment. In particular, any environmental variables that are in your 'profile's will be missing. Plan ahead accordingly, of course. See the manpages for 'crontab' for more information.

Regards!

...JRF...

[NDO]

Thank you all, I have already assign points.