topic Re: script to move files in Operating System - HP-UX

script to move files

NDO — Wed, 08 Jun 2011 11:15:59 GMT

Hi all

I have a script that move files by month, but somehow it is not working, and returns the following error:
root@nrtrde2 # ./move.sh

./move.sh: line 4: /usr/bin/scp: Arg list too long

The script is :
find /global/xnode/mcel/tap_out -type f -exec ls -lrt {} \; |
awk '{if ($6 == "Apr" && $8 != 2010) print $9}' > file_list

scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Can someone help

F.R.

Re: script to move files

James R. Ferguson — Wed, 08 Jun 2011 11:29:59 GMT

Hi:

The culprit line is this:

# scp -p $(< file_list) ...

There are too many arguments (filenames) and/or the total length is too long.

You may be forced to divide-and-conquer by either running several passes to collect files or by adding code to divide the whole file list into a series (maybe 3-10) of smaller subfiles that you then pass along to 'scp' from a 'for' loop.

You can also gain performance by changing the ':' terminator of your 'find's '-exec' to a '+'. This causes multiple arguments to be collected and passed to the pipeline, thereby dramatically reducing the number of 'awk' processes that are spawned. Your users may appreciate this.

Regards!

...JRF...

Re: script to move files

NDO — Wed, 08 Jun 2011 11:33:37 GMT

Hi

The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...

F.R.

Re: script to move files

James R. Ferguson — Wed, 08 Jun 2011 11:56:37 GMT

Hi (again):

> The issue is that I am running that find command to search for april files in 300 subdirectories.... then send them to another server...

That's not the issue. The issue is that the *results* of the 'find' are too vast to form an argument list on the 'scp' command line.

Divide-and-conquer as I said above.

You could also break your 'find' into chunks by doing something like:

# touch -mt 201104010000 /tmp/ref1
# touch -mt 201104152359 /tmp/ref2
# find find /global/xnode/mcel/tap_out -type f $ -newer /tmp/ref1 -a ! -newer /tmp/ref2 $ >> file_list
# scp ...

Wrap this logic in a loop that varies the timestamps of the reference files created and walk through the timeframe you need.

This also eliminates the majority of the forks you original code does!

Regards!

...JRF...

Re: script to move files

NDO — Wed, 08 Jun 2011 14:24:58 GMT

Hi

I have followed your advise and now I am getting issues on the "scp"

I am having a syntax error.
can you share with me your knowledge, please.

F.R.

Re: script to move files

James R. Ferguson — Wed, 08 Jun 2011 15:11:18 GMT

Hi:

> I have followed your advise and now I am getting issues on the "scp" I am having a syntax error.

Post your script. WIthout it and the actual error I can only guess ...

Regards!

...JRF...

Re: script to move files

James R. Ferguson — Wed, 08 Jun 2011 15:16:55 GMT

HI (again):

I meant to have written this block :

...
find /global/xnode/mcel/tap_out -type f $ -newer /tmp/ref1 -a ! -newer /tmp/ref2 $ > file_list
scp -p $(< file_list) ...

That is, a redirection that overwrites the 'file_list' each time though the loop, not one that appends to the contents each time :-(

Regards!

...JRF...

Re: script to move files

NDO — Thu, 09 Jun 2011 05:07:52 GMT

Hi

Sorry for the late reply:

when I run:
scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

I got the following error:
syntax error: `(' unexpected

Regards

Fernando

Re: script to move files

Dennis Handly — Thu, 09 Jun 2011 06:46:20 GMT

>I got the following error: syntax error: `(' unexpected

Nothing obvious. Perhaps you have a file with a "("? Try:
echo scp -p $(< file_list) root@10.100.48.11:/dcs/data02_loc/PROD/INPUT/TAPOUT/

Though if you aren't using a real shell, like the scummy C shell, you may get errors like this on the $() syntax. Use `` instead.

Re: script to move files

NDO — Thu, 09 Jun 2011 06:52:13 GMT

Hi

I did try that, see the o/p:

root@nrtrde2 # echo scp -p $(< file_list)
syntax error: `(' unexpected
root@nrtrde2 # echo $SHELL
/sbin/sh

And there is no special characters in file_list

Regards

Fernando

Re: script to move files

Dennis Handly — Thu, 09 Jun 2011 07:53:33 GMT

# echo scp -p $(< file_list)
syntax error: `(' unexpected

Then use quotes: echo "scp -p $(< file_list)"

># echo $SHELL

This only works if you don't invoke a different shell.
Try: ps -fp $$

Re: script to move files

NDO — Thu, 09 Jun 2011 08:20:09 GMT

Sorry Dennis!

I am lost:

root@nrtrde2 # ls file_list
file_list
root@nrtrde2 # echo "scp -p $(< file_list)"
scp -p $(< file_list)
root@nrtrde2 # ps -fp $$
UID PID PPID C STIME TTY TIME CMD
root 23099 23097 0 12:22:01 pts/1 0:00 -sh
root@nrtrde2 #

the echo just yields that

regards

F.R.

Re: script to move files

Dennis Handly — Thu, 09 Jun 2011 09:07:09 GMT

What HP-UX version are you on?
After you type escape in your real shell, what version does control-V show?

When I do that echo:
$ echo "scp -p $(< file_list)"
sh: file_list: Cannot find or open the file.

(Because I don't have file.)

About the only way that echo can produce that output is if the file "file_list" contains the string:
$(< file_list)

Re: script to move files

NDO — Thu, 09 Jun 2011 09:37:53 GMT

Hi James & Dennis

Its now working fine, what I did was to use as a second marker as file (/tmp/ref3) file with a much shorter period of time 201104052359.

Files are now being sent to the other server, but I will have create several chunks of markers in order to send all the April files.

Thank you for your help

regards

Fernando

Re: script to move files

James R. Ferguson — Mon, 08 Aug 2011 20:33:54 GMT

Hi (again) Fernando:

> Thank you for your help

If you are satisfied with the help that you received, please read the following about assigning points:

http://h30499.www3.hp.com/t5/help/faqpage/faq-category-id/kudos#kudos

Regards!

...JRF..

Re: script to move files

NDO — Thu, 09 Jun 2011 11:17:13 GMT

Yes James, I always assign points to everybody, but just one more query.

I did run the those comands on the prompt. If I put those on a script will it work?

regards

Fernando

Re: script to move files

James R. Ferguson — Thu, 09 Jun 2011 12:09:23 GMT

Hi (again) Fernando:

> I did run the those comands on the prompt. If I put those on a script will it work?

Yes. However, if you plan to run your script from a 'crontask' remember that 'cron' establishes a minimal environment. In particular, any environmental variables that are in your 'profile's will be missing. Plan ahead accordingly, of course. See the manpages for 'crontab' for more information.

Regards!

...JRF...

Re: script to move files

NDO — Thu, 09 Jun 2011 12:31:46 GMT

Thank you all, I have already assign points.