rcp or rsync in an efficient way (Script problem)

[Chern Jian Leaw]

HI

I would like to perform a remote copying over the WAN by using rcp or rsync. There's 100 filesystems which needs to be copied over the WAN. Once copied, these filesystems would need to be kept in sync(changes occuring on a source machines should be reflected and the destination machine.

This is how I've done it:
#!/bin/sh
for i in `cat file1`
do
rcp -p -r mickey.willowglen.com:$i /destination_filesys
done

OR
#cat myscript.sh
#!/bin/sh
for i in `cat file1`
do
rsync -avz --delete mickey.willowglen.com:$i /destination_filesys
done

I then invoked these script in multiples by running them at the background
#myscript > tmpFile &
But that did not improve the situation.

Is there a more efficient/better way of performing such remote copying/ or sync-up, with so many filesystems involved??

Could someone please help me out?

Thanks.

[Frank Slootweg]

You say "WAN", which is probably low speed. If so, the WAN is the bottle-neck, not the filesystems/disks, so you should do the remote copy operations sequentially (one after the other) instead of pin parallel.

Famous words: If you are in a hurry, you should take it easy.

[U.SivaKumar_2]

Hi,
Better use FTP for initial file transfer and rsync for updating the files.

regards,
U.SivaKumar

[Chern Jian Leaw]

Frank, Siva,

I'm expecting 800GB of filesystems to be transferred using the WAN, and later perform the incremental updates using rsync.

I ran the script in parallel by feeding the script with different input files i.e
#cat inpufile_1
/fs10/my_circuits/
/circuits_project/simulations/

#cat inputfile_2
/fs11/xxx/
/fs13/yyy/

#myscript inputfile_1 > tmpFile_1 &

#myscript inputfile_2 > tmpFile_2 &

Are there any other ways which I can exploit the info on the threads/number of rsync/rcp/ftp processes running, then invoke more of such process if the number of running process has not exceeded the threshold?

Thanks.

[U.SivaKumar_2]

Hi,
I recommend not to start multiple sessions for multiple files as this will cause heavy congestion in your WAN and will slow down ( will cause more delay for the packets )the
realtime traffic services like telnet , rlogin
etc. Telnet will hang for example.

Only way is to do sequentially even if it takes
a night. 100files/1 night

regards,
U.SivaKumar

[Chern Jian Leaw]

Hi does anyone have any recommendations on how the script could be improved in such situations?

I did the following, where file1 contains the list of 100 filesystems:

for i in `cat file1`
do
numProcess=`ps -ef|grep rsync|wc -l
if [numProcess < 10 ]
then
rsync -avz mickey.willowglen.com:$i /destination_filesystems
else
sleep 300
fi
done

The script would copy or incrementally update all 100 filesystems, but with 10 filesystems at a time.

Could anyone help me out on making improvements to such scripts, at the same time achieving efficient throughput time over the WAN?

Thanks

[Frank Slootweg]

As we mentioned, we, U.SivaKumar and I, advise to do it one filesystem at a time, so please do not ask us how to do it for multiple filesystem at a time.

What is the speed of the WAN? Please clearly specify bits versys bytes, etc..

[Chern Jian Leaw]

Frank,
The speed of the WAN is 0.5GB(GigaBytes)/hr. Hence, if there's 50GB of filesystems to be copied over the WAN, it would take 100 hours i.e approx 5 days to copy stuffs over.

Any ideas?
Thanks

[A. Clay Stephenson · Accepted solution]

Given the speed of your WAN, this is a pointless exercise using rsync. You can probably shave some time by compressing the data before transmitting but really your fastest data tranfer method is a media transfer via Federal Express.

[Philip Chan_1]

My suggestion for you would be using rsync cos' it is very efficient for file transfer. I've tested rsync against ftp and found that rsync is significantly more efficient than ftp. In addition, the other advantages of rsync I could think of are,

1. It will not send un-modified files. Meaning that if only 2G out of the 800GB data were updated, then rsync will send out these 2G updated files only.

2. Rsync can perform compression before sending out files. This is good for slower networks.

Just try transferring a 10GB using rcp, ftp and rsync then you'll see the difference in raw transferring speed !!!

[harry d brown jr]

You stated this:

The speed of the WAN is 0.5GB(GigaBytes)/hr. Hence, if there's 50GB of filesystems to be copied over the WAN, it would take 100 hours i.e approx 5 days to copy stuffs over.


But earlier you stated you expected 800GB.

That's 16 TIMES greater than 50GB. Using that figure it would, based upon your calculations, 80 days to make the initial transfer. Now, if this is true, the data change that could occur, might cause the transfer of data change to be exponentially larger than the initial transfer.

Imagine this: You are transfering a file while it is being modified. How will this change me checked, and the data transfered?

I'd suggest you consider, although expensive, purchasing EMC and use srdf to accomplish this.

Unless you are into writing software that would revolutionize the world, I think you need a "hardware" solution.

live free or die
harry