rsync and large file support.

[Aharon Chernin]

I have been fooling around with this problem on and off for the past few months.

rsync 2.5.5 as well as the older versions support large files (>2 gigs)..

For the life of me I cant get large file support to work in rsync. Rsync reports that largefile support is compiled in, but yet wont copy files over 2 gigs.

Any ideas? I think I have recompiled rsync now atleast a 100 times trying different compiler options.. lol

rsync version 2.5.6cvs protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
<>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
no IPv6, 32-bit system inums, 64-bit internal inums

[harry d brown jr]

What OS to what OS are you using? And if HPux, do you have largefiles turned on?

live free or die
harry

[Aharon Chernin]

Sorry, I was vague :) Too busy pulling my hair out.. lol

HP/UX 11.0 ..
The filesystems I am copying too and from, have largefile support and are mounted with the -o largefiles options.

Is that what you mean by largefile support turned on?

[harry d brown jr]

Why not use this:

http://hpux.cs.utah.edu/hppd/hpux/Networking/Admin/rsync-2.5.5/

although not 2.5.6?


live free or die
harry

[harry d brown jr]

Yes to largefiles.

Try the precompiled version.

live free or die
harry

[Aharon Chernin]

I just tried using the binary you posted up.. And to speed up the test of that binary version of rsync, I copied a 2.5 gig file from one place to another on the same box.

It failed just like my compiled versions have done. I have a ton of hp/ux 11.0 boxes and none of them seem to be able to do this properly :(

Here is a 2.5 gig file test with the rsync link you provided.. btw it's the same error 2.5.6cvs gives.

aurora:/premdordev/MP5.BCV # rsync --progress -arvv IN.PAST.HIST.COPY aurora.premdor.com:/premdordev/IN.PAST.HIST.COPY
opening connection using remsh aurora.premdor.com rsync --server -vvlogDtpr . /premdordev/IN.PAST.HIST.COPY
building file list ...
expand file_list to 4000 bytes, did move
1 file to consider
IN.PAST.HIST.COPY
rsync: error writing 32768 unbuffered bytes - exiting: Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(463

[harry d brown jr]

rsync: error writing 32768 unbuffered bytes - exiting: Broken pipe

It looks like it's keeping count of "blocks" with a 2 byte integer.

Up the SIZE of the blocks per second?

--bwlimit=KBPS


live free or die
harry

[Matthew Carlson]

Here is the fix for the HP-UX 11.00 and 11i (11.11) large files issue with rsync 2.5.5.

You just need to compile it 64bit. I just used the DA2.0w CFLAGS option.

export CLFAGS="-Ae +O4 +Onolimit +DS2.0 +DA2.0w"

That is it.