Multi tasking in unix shell

[David L Brewster]

Hello

We periodically transfer files from our production box to our DR box throughout the day. Each file is about 50M and it had been taking about 3-4 seconds per file to rcp to DR.
Now our DR site has moved to Seattle (across the US for me). The same process is now taking 3 minutes per file. I wasn't expecting that much of a delay. Our network people say that it is not a bandwidth issue and I should try transfering, say, 5 files at a time rather than one at a time. Transfering 5 files at the same time should also take 3 minutes. The only way I have thought to do this was to put 5 rcp commands in the background and then monitor the progress.
Is there a better way?
Please withhold security concerns about using rcp!

[Dennis Handly · Accepted solution]

>Our network people say that it is not a bandwidth issue and I should try transferring, say, 5 files at a time rather than one at a time.

There are two ways to read that. Do what they said or change to batch up N files and send them at once. But this assumes that the delay is due to finding the files.

Of course doing a tar and somehow doing the untar on the other end is doable.

>The only way I have thought to do this was to put 5 rcp commands in the background and then monitor the progress. Is there a better way?

If your bottleneck is the CPU/DISK, doing it is parallel won't speed up your network traffic, unless you have a CPU to spare. If you are disk limited, you need to fix that.

[Steven E. Protter]

Shalom,

This is a network latenecy problem pure and simple. Move the DR center further away and the latency gets worse.

A simple ping command run to the old DR site will how less ms delay than the new.

You can ask your Internet provider to check the line and see if the issue can be addressed.

Probably not though. If you are willing to put a network device called a WAN accelerator at both ends of the connection (Base price $40,000 per unit) you can significantly improve the performance of your current system.

rcp is the fastest and least secure way of doing the transfer. Using a different program or script methodology will probably not help.

You could also buy a large bandwitdh from the Internet provide. That might be very expensive as well.

Surely taring the files up and compressing them before transfer should help the performance issue somethat. It is possible however that due to latency you are exceeding your line capacity and there is nothing you can do to fix it that doesn't cost a lot of money.

If you use the unix shell to tranfer files in the background the nice value is automatically 24 versus 20 for a foreground process. In summary, it will make things take longer.

witholding my concern that its not smart to send critical data across the country unencrypted.

SEP

[David L Brewster]

Dennis:
Thanks. Going to try tar and then compress.
Good idea.

Steven:
Thanks for the reminder on the nice value for background processes. Also going to mention your comments to my director...and claim them as my own!

Many thanks to you both!
Happy New Year.

[James R. Ferguson]

Hi David:

Here's a good way to measure your bandwidth *without* involving your filesystems:

...Open a BINARY FTP session and do:

# put "| dd id=/dev/zero bs=1024k count=1024" /dev/null

...this would transfer 1GB without involving any filesystems.

Regards!

...JRF...

[Dennis Handly]

>SEP: If you use the unix shell to transfer files in the background the nice value is automatically 24 versus 20 for a foreground process.

If this is a problem you can change it by: set +o bgnice

[David L Brewster]

Thanks to all!

[Dennis Handly]

>Going to try tar and then compress.

gzip does a better job.