Fastest file transfer between two HP-Unix sean rvers in same network

[Mangal Pandey]

I need to copy lot of small files between two HP box. I have an application which call the RCP scripts to copy these files but RCP is taking time and thus slowing the speed of application.
Can anybody suggest me any other file transfer techniques which will be the fastest one.
---Files are small (But there will be thousands of files at the same time to copy and a number of concurrent application will be invoked to copy number of files at the same time).
---Both servers are in same network.
---Need the fastest way to send file from one server to another.

[Dennis Handly]

You may want to consider tarring up the files and if large, gzip, then rcp that tarball. Of course you would have to untar it on the other side.

I've seen others that said ftp was faster??

[Steven Schweda]

There's not enough information here to give a
reliable answer, and it's probably the wrong
question.

Why copy the files? Why not use NFS?

Why use "thousands of files" for any kind of
data sharing?

How often do the files change (and need to be
re-copied)? Who makes the files?

Why ask for the best way to implement a bad
solution to a problem?

[Mangal Pandey]

Hi Steven,

There are two application running on two different servers. One application will generate the file and other has to process the files. First application generates all new files and in thousands of files per minute and second application has to process those files locally.

Can you suggest any approach for second application to handle the files faster assuming all time is taken in transferring files from on server to another.

you are telling something like NFS and i can implement this as well if this is the best way to do.

[Steven Schweda]

NFS would let a program on the second server
see (normally read/write) files which are
stored on the first server, without anyone
explicitly copying them.

I'd guess that NFS would be faster than using
"rcp" to copy the files, but an actual test
would be more reliable than my guess. It
would certainly be simpler. Some of the
speed depends on how the systems are
connected.

There are also various kinds of shared
storage available in the world. In the VMS
world, cluster members share disks with even
less effort than using NFS.

[Emil Velez]

I got a crazy idea.. If the boxes are HPUX 11.23 and you have ServiceGuard why not just use a cluster file system between the systems so the files are on both systems concurrently. CFS was released as a add-on to ServiceGuard and it uses Cluster VXVM and I have also used it for oracle data files with RAC but that is not needed if you just want a shared file system.

[Jov]

Hi,

I'd suggest either NFS or Samba/CIFS depending or what you need and comfortable in doing.

Go with NFS for its only and will likely stay with Unix access to the files, but if you think there might be a Windows element down the road, then setup Samba/CIFS on your 1st HP-UX server that is producing the files.

Have a read of the following post for details
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=3368

Jov

[Mangal Pandey]

Based on the response above we have three choices NFS, Samba and CFS.

we can discard samba as no any windows interaction is there its all HP-UX servers.

Now for NFS, i read somewhere that it may affect the performance and sometime it is even slower then RCP. Is this true?

Now finally CFS is new to me and i'll definately google about it but a quick query here is that, will it affect the performance of application if read and write are done to CFS drives.

Thanks all for your responses.

[Jov]

NFS is your best and quicks way to setup. With CFS, you'll need an additional license where as NFS does not.


Jov

[Steven Schweda]

> [...] it may affect the performance [...]

Everything you do will affect the
performance.

I suspect that an actual test in your
situation will be much more reliable than any
estimates you'll be able to get here. You
already know how well rcp works. A test using
NFS should be pretty easy to arrange.

[Bill Hassell]

For large files, ftp is the fastest. But for very small files (less than few dozen megs) the problem has very little to do with the speed of the link, but the filesystem overhead in both ends of the transfer. rcp, ftp, scp, all of these tools use the the local and remote filesystems and that's where most of the delays will occur -- not a networking issue.

That's why NFS is the appropriate tool. Now keep in mind that NFS is realtime, so as a file is being created on the first machine, it will also be created on the other machine. So you will have to provide a semaphore to notify the second machine that the file creation is complete.

[Steven Schweda]

> [...] as a file is being created on the
> first machine, it will also be created on
> the other machine. [...]

NFS does not cause a file to be created in
multiple places at the same time. A more
accurate way to think of it would be: as a
file is being created on the server, it will
be visible (or accessible) from the client.

> So you will have to provide a semaphore to
> notify the second machine that the file
> creation is complete.

Which is not much different from knowing when
you can copy a file using rcp.

[Andrew Young_2]

Hi.

There are a number of issues you need to consider here.

Will the files be remaining where they are? Or will they be moved? Once these files are created how long are they being kept for before being removed? What is an acceptable time delay between moving the file and it being available to the second server?

NFS is probably the best solution but be careful not to flood the network. Maybe create a private network if this is a concern.

Also you do not want to copy open files.
So I would create the file in one directory, once it has been created then I would move it to a second directory on the same filesystem and and have that directory exported for NFS. Then access the files directly off the other server rather than copying them.

The problem with many small files is that opening and closing them creates a lot of overhead.

Maybe the following configuration changes will help. Upgrade the memory on the server and adjust the dbc_max_pct kernel parameter upwards so the documents are cached which will speed things up.

The other option would be to combine files together in a zip volume. The data over the network will be compressed, there will be fewer inodes open and the receiving server can do the processing/file handling. Note that some versions of zip have a limit on the number of files they can contain (32768 I think). This of course depends on how smart the app on the other end is. If it expects a flat file and can't be modifed via a script then this is not an option.

I would not use rcp or scp as both of these have high transaction set up costs.

CFS is an option if both servers are on the same SAN.

If a small delay is acceptable and the files are going to stay around for a while looking at a mirroring solutions like rsync could be an option too.

HTH

Andrew Y

[rick jones]

Is the overehad of file creation and file processing really high enough to require two systems? Why not one larger system and then not have to worry about the transfer in the first place?

FWIW, when Bill said that the file was created in the two machines at the same time, I believe he meant to imply that as the file is created on the one system, it will be visible to the other, so there would need to be some mechanism for the processing system to know when the file was complete.

[Alzhy]

Neither of the solutions provided so far I think will fit Messr. Pandey's dilemma.

My suggestion is to set up vxdump/vxrestore jobs to transfer those zillion or so files from ServerA to ServerB. Any other method will entail overheads. Here's how you would do it:


From ServerA where those zillions of tiny files are generated for transfer/processing to ServerB, do:

vxdump 0uf - /sourcefs | ssh ServerB "(cd /targetfs;vxrestore rf -)"

Then subsequent transfers you can use "levels" so only newly generated files are transferred, ie:

vxdump 9uf - /sourcefs | ssh ServerB "(cd /targetfs;vxrestore rf -)"



Or you can employ "rsync" after your first vxdump/vxrestore if you prefer and transfer only the differences between source and destination - but I would still bet for vxdump/vxrestore anytime.

Hope this helps.