Large data transfer over network?

[Dan Alexander]

I have over 200GB of data to transfer between 2 systems. I would love to do this over a "private" LAN connection, but I don't have any free lan cards in one system. So, using my existing production lan, what would be the best method for moving this data?
NFS, FTP , RCP, cpio, tar, etc...? Anyone have any thoughts? Thanks!

[A. Clay Stephenson]

FTP will probably give you the best network results. Yo would probably be best served by compressing the files before sending. NFS will have (by a fairly wide margin) the greatest overhead. If you have some spare disks then by far the best method is a vgexport/vgimport. Avoid tar and cpio because (unless you use the Gnu version) have a 2GB limit.

[Paul Sperry]

By far FTP will give the most efficient and reliable results. Also try to do this at night if you not a 24hr shop.

[James A. Donovan]

If you're worried about bandwidth, then take a look at rsync.

http://rsync.samba.org

It has an option to limit the amount of bandwidth used for the transfer.

[Steven E. Protter · Accepted solution]

Being the contrarion, and wanting to do it secure if not quite as fast I'd use scp.

Install this.
https://payment.ecommerce.hp.com/cgi-bin/swdepot_parser.cgi/cgi/try.pl?productNumber=T1471AA&date=

I like scp because it gives a nice progress bar as it goes. You can see transfer rates and network stalls as well. That is an advantage over ftp. Because encrypted keys are exchanged there are no clear text passwords, but this adds a few seconds to overall transfer time.

NFS will tend to have problems if your switch lan network configuration isn't perfectly matches and not set to autonegotiate.

rcp is not a secure protocol and simply should not be used.

tar can go up to 8G but I still wouldn't use it. HP has a good patch for tar but its not relavent in this context.

ftp will work just fine, I just hate using anything that transmits passwords in clear text across the network.

I work for a Jewish organization and we're kind of paranoid.

SEP

[Scott Van Kalken]

I know it might seem obvious - but how about a tape?

No load on your lan cards.

[Dan Alexander]

Thanks All,

Steven - I will take a look at what you mentioned...

Tape? Nah! Don't have any tape drives...

Because I moving hundreds of directories, I installed xftp, and am testing it out..(because it copies dirs recursively!) So far, xftp seems pretty cool..

Thanks for all the good info..

Dan

[Gerhard Roets]

On a side.

Ncftp client has been ported to HP-UX. It is alos nice and user friendly and text based ;)

Just a thought
Gerhard

[Steven E. Protter]

Now that I am at work, I can upload my (borrowed) CIFS/Samba cookbook to exchange public keys and have ssh, scp sftp work without passwords.


Attached.

SEP

[rick jones]

Scott brings-up a good point. Never underestimate the bandwidth of a station wagon full of tapes (replace station wagon with your favorite conveyance here). and i'm even a networking guy :)

[Chris Vail]

I'm going to concure with Mr. Protter about scp, and also recommend rcp. Both of these use UDP packets, which on a simple, point-to-point connection are considerably simpler ways to transfer information. They're easier on your bandwidth and easier on the CPU, hence faster throughput.
Race the two: try ftp'ing a large file between the two systems (use timex), and then try it with rcp. I think you'll find that rcp is 20-30% faster on most systems and networks.
I prefer scp, as it is secure and encrypted. Its the ONLY utility we use here now. Its a little slower on initial connection (4-20 seconds) but once the keys are exchanged, its just as fast or faster than any other. Plus, you get that real cool percentage indicator that you don't get with the other utilities.

I seem to be doing this 2-3 times a week: attaching my "Using Secure Shell" document. Follow the steps CAREFULLY, and you'll set up scp nicely.

Chris

Chris

[rick jones]

chris - are you _certain_ that scp and rcp use UDP? my impression lo these many years is that rcp at least uses TCP exclusively. That is transfers faster than FTP could be a difference between rcpalways being a "binary" transfer and some FTP's defaulting to ASCII and so having to scan every byte. It could also have something to do with the socket buffer sizes used by default in the specific implementations of the file trasnfer utilities. For example, FTP on UX defaults to a 56KB socket buffer and thus window.

[Chris Vail]

Rick--I'm fairly sure about rcp using UDP packets.....I have a poster up on the wall that covers this. rcp is a subset of the rpc services (portmap, rpcbind, etc) and all of these use UDP. I tried googling up something definitive, but don't have much time. It looks like rcp uses UDP in Sun, and TCP in some others, so maybe we're both right. This was true on an old AIX box I used to sysadmin. However, my /etc/services file doesn't have an entry for rcp, so who knows?


Chris

[Jeff Schussele]

Chris/Rick,

rcp along with remsh, rdist & rcmd are all functions of remshd.
If you check your /etc/inetd.conf for the line ending in remshd - you'll see:

shell stream tcp nowait root /usr/lbin/remshd remshd

So it's definitely TCP for HP-UX. Cross reference that to the /etc/services file & you'll see it's 514/TCP to be exact.

Rgds,
Jeff

[rick jones]

i've always understood rcp to be part of the BSD remsh (rsh) rexec family, not part of sun's onc rpc family.

i'll put lunch on rcp and scp always using TCP :) tusc and such should show that by showing the parms to the socket() calls made.

btw, what is the source of the poster?

[Jeff Schussele]

I'd say that's a very safe bet.

Later,
Jeff

[TwoProc]

I'm going to give you an answer that has makes an assumption that the others don't ( that at some time, you have full use of the bandwidth ). If this is not a correct assumption, then so be it.

First cd to the directory and machine that you want to copy from. Now, run the following:
tar cvf - | remsh destination_machine "( cd /destination_directory ; tar xvf - ).

This will allow you to see your copy, as well as move files larger than 2 Gig over to the destination box). It has all of the disadvantages of lax security, full hogging of the network bandwidth, etc.
But makes copying files from one system to the other painless because it correctly preserves a) links (hard and symbolic, b) permissions, and c) ownerships unlike rcp which doesn't manage any of these well.

[TwoProc]

I'm going to give you an answer that has makes an assumption that the others don't ( that at some time, you have full use of the bandwidth ). If this is not a correct assumption, then so be it.

First cd to the directory and machine that you want to copy from. Now, run the following:
tar cvf - | remsh destination_machine "( cd /destination_directory ; tar xvf - )".

This will allow you to see your copy, as well as move files larger than 2 Gig over to the destination box). It has all of the disadvantages of lax security, full hogging of the network bandwidth, etc.
But makes copying files from one system to the other painless because it correctly preserves a) links (hard and symbolic, b) permissions, and c) ownerships unlike rcp which doesn't manage any of these well.