Transfer very large files across servers.

[Gulam Mohiuddin]

I need to regularly move/copy very large data files (each about 4GB) between two HP-UX servers every night.

Most probably I will be using unix "rcp" command to accomplish this.

I need some hints to speed up this transfer:

1.Do I need to tune any HP-UX Kernel parameter to make it faster by reading bigger chunks of blocks from file systems etc?

2.If I compress files at host “A” and copy to host “B” and then uncompress it, does this will speed up the transfer?

3.Can I transfer files in parallel?


Thanks.

Gulam

[Steven E. Protter · Accepted solution]

Shalom Gulan,

1. I don't think kernel tuning will help. Its a network bandwidth issue.

2. Yes, compression does speed the transfer but it requires diskspace to compress and the time saved is exceeded by the time it takes to compress. Still the file transfer itself will be faster. gzip had the best compression last I checked.

3. Yes, but it will probably take just as long. Network bandwithc is the issue.

If the two machines are on the same network, see that they have gigabit networking and a good gigabit switch between them.

SEP

[Jeff_Traigle]

See Steven's comments about 1 and 3. I do not agree that compression will necessarily help speed things up, however. It might. It might not. It depends on the network between the systems. In running some tests on various platforms a couple of jobs ago, I found that once you hit 100Mbps full-duplex connectivity between systems, the overhead of doing compression actually made the transfer slower. (I was doing the "tar | remsh" method so everything was done in memory.)

[Bill Thorsteinson]

1. This is not likely something you can tune
the kernel for.

2. Use scp and you can compress the data on
the fly. Data can be compressed on the wire.

3. Transferring files in parallel may cause
problems with overloading the receive buffers
on the target machine.

If these are not new datafiles, but mearly
updates to existing files, consider using
rsync. It will merge the changed and new blocks into the destination file. It uses
checksum comparisons to identify changed
blocks. It usually runs over ssh, and
compression level can be tuned. You may
want to use it even if the files are new.

[Dave Olker]

Hi Gulam,

I am completely biased because I've worked with NFS for years, but you could use NFS to do this.

I've spent the past few years working on NFS performance tuning issues and I've been able to get NFS/TCP to run at wire speeds for both reading and writing using Gigabit interfaces. In fact, during a recent engagement I was able to get many, many GigE interfaces running at wire speed with NFS/TCP traffic.

Just a suggestion, since NFS ships with all of our systems and you can use standard "cp" commands to move files around. NFS certainly supports parallel transfers via multiple cp commands.

Regards,

Dave

[Bill Hassell]

The fastest transfer method is ftp. rcp and scp are not tunable and do not optimize transfers based on the wire speed like ftp. NFS introduces filesystem codes into the data exchange, thus reducing the data speed. ftp will adjust packet sizes to optimize the transfer as well as transmit multiple packets before requiring an acknowledgement. This is extremely important if the link is not a LAN but a WAN. Most WANs have a long turnaround time from transmit to receive so short overhead packets such as NFS filesystem protocol can really slow down transfers for short files.

[Arunvijai_4]

Hi Gulam,

1) Kernel tuning won't help in this case as networking speed related with external parameters like Bandwidth, Traffic, etc..

2) You can use gzip or compress to reduce the file size. This time you will have reduced file with same content and it will be lesser time than the previous one(uncompressed)

3) Yes, You can transfer files in parallel.

Try to use SCP or SFTP since it will provide better performance than RCP.

-Arun

[Yogeeraj_1]

hi gulam,

i would go for FTP too.

Don't forget to set the appropriate mode for the transfer: Binary or ASCII

You can of course do it in parallel.

if you server have multiple network interfaces, you can also benefit from that.

khope this helps too!

kind regards
yogeeraj

[Sยภเl Kย๓คг]

Hi,

What is the type of connectivity b/w those two servers, is it LAN or WAN. In both cases I will recommend ftp, because it has less overheads. You can compress the files using compress command, not gzip and then ftp it, sftp is also preferable. If I'm not wrong gzip wont supports files aboove 2TB. I am not sure any patching is available on that.

If it's ur regular activity and u can invest some money, go for some data compression boxes b/w the machines, which will reduce your headache of data compression and decompression at unix level.


Regards,
Sunil

[rick jones]

It all depends:

*) How fast (bandwidth) is the network between the two servers?

*) How "far apart" (ping times) are the two servers?

*) What CPUs do you have in the two servers?

*) How much memory is there on the target server?

*) How fast are the disc(s) on the source and the target systems?

*) Are the different files going to the same or different filesystems/discs?

[Geoff Wild]

You say regularly - so I would use rsync or rdist

Here's what I do:

created /app/admin/drp

created script to update files tn DR server called update-binaries.sh

created file to store what to update called distfile

/app/admin/drp # cat update-binaries.sh
#! /bin/sh

# Keep the DRP copy of the vgpadm up-to-date.
# Currently the files are in:
#
# /home/vgpadm
#
# See the rdist(1M) distfile for a list of exclusions.

DRPDIR=/app/admin/drp
DRPHOST=svr032

mount | grep /home > /dev/null 2>&1
if [ $? -eq 0 ]
then
( su - vgpadm -c "rdist -f $DRPDIR/distfile vgpadm"; ) 2>&1 |\
tee $DRPDIR/drp.log 2>&1 |\
mailx -s "VGPADM DRP rdist output" gwild@mydomain.com
fi

/app/admin/drp # cat distfile
VGPDR = ( svr032 )

#
# File systems to be copied over to the DR host.
VGPADM = /home/vgpadm

vgpadm: ( ${VGPADM} ) -> ( ${VGPDR} )
install -R -w ;
except ${VGPADM}/logfiles;

Add to cron:

# Copy vgpadm across to the DR site.
05 01 * * * /app/admin/drp/update-binaries.sh

Rgds...Geoff