looking to improve remote replication process.

[Manuel Contreras]

looking for solution to replicate 1.5GB files to a remote location...
Currently, this process looks like the following:

move 1.5GB files into a staging area.
compress files.
rsync files to remote server.
remove compressed files.


I have performed some timings, and compress seems more efficent over gzip.

Is there an efficient way to move and compress in a single step?
wondering how to improve process listed above...
thanks

[IT_2007]

It depends on the bandwidth of your network to remote server.

If you have enough space on the local server then you can compress before you copy to remote server.

you can use scp which is secure copy over the network.

[spex]

Hi Manuel,

You could use a named pipe:

mkfifo cmp_pipe
sleep 5
nohup tar cvf - /path/to/files/* | compress -c - > cmp_pipe &
# copy to remote server...
wait
rm -f cmp_pipe

PCS

[Manuel Contreras]

I am gathering some stats on sending compressed vs non-compressed file to remote box.
It may make more sense to compress at remote box...

Will post stats on copy later, but the compression stats are as follows:

# time /usr/bin/compress bigFILE

real 2:29.3
user 1:46.6
sys 23.2

# ll
-rw-r----- 1 root sys 549384653 Sep 20 10:10 bigFILE.Z

1566427136-549384653
1017042483 amt. compressed
1017042483/149 sec

6825788.47 per sec


gzip file:

# time /usr/local/bin/gzip bigFILE
real 5:02.2
user 4:21.2
sys 22.5
# ll
total 778816
-rw-r----- 1 root sys 398751664 Sep 20 10:10 bigFILE.gz

1566427136-398751664
1167675472 amt. compressed
1167675472/302 sec

3866475.07 per sec

[Jeff_Traigle]

Compress is more efficient in time to compress, but, as you can see, gzip does a better job of compressing the data. Testing I did a couple of years ago on a project showed it was best to compress (I think I had to opt for compress because the process had to work on older 10.20 systems also, which did not have gzip installed) before transferring files unless you had at least 100Mbps full-duplex bandwidth between the systems. At that point, the cost of compressing the files negated the benefit of transferring less data.

[Manuel Contreras]

since rsync performes compression in-flight, it likes uncompressed files:

stats on sending compressed file:
(your milage may vary :D )

551499.52 bytes/sec
total size is 549384653 speedup is 1.00

real 16m35.83s
user 0m10.71s
sys 0m30.54s


uncompressed file:

1566427136 100% 2.08MB/s 0:11:58 (xfer#1, to-check=0/1)

sent 42 bytes received 397960981 bytes 552340.07 bytes/sec
total size is 1566427136 speedup is 3.94

real 12m0.30s
user 0m56.81s
sys 0m28.56s


How about the best of both worlds?
rsync uncompressed file, and gzip on remote box...sounds good to me :D