topic Re: run multiple shell via script to copy file in Operating System - HP-UX

run multiple shell via script to copy file

RiclyLeRoy — Tue, 23 Jan 2024 04:30:21 GMT

I need to copy 5 file from server to another one, I scheduled this copy by script in cron but I want to parallelize copy opening multiple shell.

What do you think ?

Can you suggest command to invoke a new shell for every file copy without to wait copy ends ? My target is to get 5 shell processes which are copying files.

Re: run multiple shell via script to copy file

Steven Schweda — Mon, 22 Jan 2024 18:33:26 GMT

> [...] copy by script [...]

Actual copy command(s)? (cp? scp? ftp? ...)

> [...] I want to parallelize copy opening multiple shell.

Why, exactly?

> Can you suggest command to invoke a new shell for every file copy
> without to wait copy ends ? [...]

   Append an ampersand to your file-copy command? For example, change:
      cp source destination
to:
      cp source destination &

You might want to consult the documentation for your shell to learn
how to determine when all the background jobs have completed.
(Something you don't need to worry about if the operations are
sequential.)

> What do you think ?

I suspect that you're making a simple job into a compicated job with
little if any benefit.

> [...] I want to parallelize copy opening multiple shell.

Why, exactly?

If there's some communication bottleneck between these servers, then
I would not expect performance to improve much if you try to push more
data through that bottleneck by having multiple processes try to do it
simultaneously. The bottleneck will still be there, no matter how many
processes are fighting over it.

Have you tried manually running these copy operations simultaneously
(in separate interactive shells)? Did you see any real benefit?

> [...] My target is to get 5 shell processes which are copying files.

I wouldn't expect five (or ten) background jobs to cause any special
problems. (Or provide any significant benefit. But only one of us (at
most) knows what you want to achieve by doing this.)

Re: run multiple shell via script to copy file

RiclyLeRoy — Thu, 25 Jan 2024 21:22:16 GMT

>Actual copy command(s)? (cp? scp? ftp? ...)

scp

>Why, exactly?

Oracle exported 5 files by script and immediately I'd like to start scp copy of these files to remote server inside the same export script.

I thought to ease job making to start all the copies in the same time but I think It's the same result if I start copy sequentially.

What do you think ?

Re: run multiple shell via script to copy file

Steven Schweda — Fri, 26 Jan 2024 01:27:33 GMT

> What do you think ?

If the files are created sequentially, then copying them as they are
created might make sense. The actual benefit would depend on the time
required to create a file, the time required to copy a file, and how
much interference there is between the create and copy operations (how
much copying one file slows creation of the next file).

If creating a file is slower than copying it (and copying a file
doesn't slow creation of the next file much), then I'd expect copying
multiple files to be about as fast as copying one (the last one).

If creating a file is faster than copying it, then I'd worry more
about a communication bottleneck making simultaneous copying no (or not
much) faster than sequential copying.

Knowing nothing about the time needed to do any of these tasks, I'd
need to run the experiment to see how big the benefit might be.

> Append an ampersand [...]

The relevant topic in the shell documentation seems to be "job
control".