rsync with many files on HP-UX 11i malloc error?

[A. Daniel King_1]

Hi, folks.

I've started getting these errors from rsync, and any help would be appreciated:

>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at
>util.c(115)
>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at util.c(115)
>ERROR: out of memory in string_area_new buffer
>rsync error: error allocating core memory buffers (code 22) at util.c(115)

The file system transfer in question then fails. After Googling and looking at the man pages, I have surmised that this is likely a malloc/out of memory error. From the man page:

> RERR_MALLOC 22
> Error allocating core memory buffers

I'm using version ...

>$ rsync --version
>rsync version 2.5.6 protocol version 26
>Copyright (C) 1996-2002 by Andrew Tridgell and others
><>
>Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
> no IPv6, 32-bit system inums, 64-bit internal inums
>
>rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and
>you are welcome to redistribute it under certain conditions. See the
>GNU General Public Licence for details.

HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred have many files. I'll be testing further with more verbosity, but I know that the files to be transferred would be in the 300,000 count range. The actual quantity of data is fairly small (<20GB).

I also see that ...

"As a rule of thumb you should expect rsync to consume about 100 bytes per file in the file list."

That would make some 30,000,000 bytes, or ~30MB, which is not much on our system. Could there be other issues? I also see that there is an HP-UX compile time option (-Ae from http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4). I suspect that this applies to the HP ANSI compiler, not gcc as I am using, but I do not know if this is the cause.

I see some memory issues in the TODO list with my distribution, but I'm not familiar with the fix(es). It perplexes me that the site above references 80MB transfers - over twice the size of my current issue ... oh wait, my test is back (with double-v) ...

receiving file list ...
expand file_list to 4000 bytes, did move
expand file_list to 8000 bytes, did move
expand file_list to 8000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 16000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 32000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 64000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 128000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 256000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 512000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 1024000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 2048000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 4096000 bytes, did move
expand file_list to 8192000 bytes, did move
expand file_list to 8192000 bytes, did move
expand file_list to 16384000 bytes, did move
expand file_list to 16384000 bytes, did move
expand file_list to 32768000 bytes, did move
expand file_list to 32768000 bytes, did move
ERROR: out of memory in receive_file_entry
rsync error: error allocating core memory buffers (code 22) at util.c(115)

This looks like one of those pesky sizeof-type issues ... Would a 64-bit compile help? Are there other configure/compile options which should help? Google and the rsync list archives do not return anything on this issue specifically.

Any direction would be appreciated, even where to dig or specific cvs information ... thanks in advance!

[Srinivas_35]

Try to compile rsync in 64bit mode and you need to have gcc installed on your machine. Also check you are running 64bit by:

getconf KERNEL_BITS

If this doesn't work out then you can try rsync with earlier version 2.5.5 or 2.5.4

[Ashwani Kashyap]

Try increasing the kernel parameter maxdsiz (for 32 bit process ) or maxdsiz_64 ( for 64 bit processes ) .

[A. Daniel King_1]

maxdsiz_64 is already very high;

I have realized that the number of files does not even come close to the number of _links_, which are many more, and do count in the memory equation. I have broken down my list of hierarchies, per discussion on the rsync mailing list, and this is probably how things will remain.

Thanks for the replies!

[A. Daniel King_1]

The rsync 2.6.2 upgrade helped.

[Bill Hassell]

rsync is a 32bit app so the kernel parameter is maxdsiz. The default is a lowly 64megs, so you are probably hitting that limit. Change it to 500 megs and you should be fine. Also make sure that ulimit -a shows memory=unlimited in your environment.

[A. Daniel King_1]

maxdsiz is much larger than 64MB.

Thanks for the input, all.

[Geoff Wild]

Just an idea, can you try this with rdist instead?

rdist(1) rdist(1)

NAME
rdist - remote file distribution program

SYNOPSIS
rdist [ -bhinqvwyMR ] [ -f distfile ] [ -d var=value ] [ -m host ]
[ label... ]

rdist [ -bhinqvwyMR ] -c name... [ login@]host[:dest ]

DESCRIPTION
rdist facilitates the maintaining of identical copies of files over
multiple hosts. It preserves the owner, group, mode, and modification
time of files if possible and can update programs that are executing.


Rgds...Geoff

[A. Daniel King_1]

rsync is optimized for minimum network traffic. Items already present will not be recopied.

Does rdist do this?

[Geoff Wild]

Yes - rdist does the same.

I run it from cron to sync home directories for DR.

# Copy myadm across to the DR site.
05 01 * * * /app/admin/drp/update-binaries.sh

# cat /app/admin/drp/update-binaries.sh
#! /bin/sh

# Keep the DRP copy of the myadm up-to-date.
# Currently the files are in:
#
# /home/myadm
#
# See the rdist(1M) distfile for a list of exclusions.

DRPDIR=/app/admin/drp
DRPHOST=svr032

mount | grep /home > /dev/null 2>&1
if [ $? -eq 0 ]
then
( su - myadm -c "rdist -f $DRPDIR/distfile myadm"; ) 2>&1 |\
tee $DRPDIR/drp.log 2>&1 |\
mailx -s "MYADM DRP rdist output" gwild
fi

# cat distfile
MYDR = ( svr032 )

#
# File systems to be copied over to the DR host.
# Don't use -R in install - so as not to remove files on destination host
MYADM = /home/myadm

myadm: ( ${MYADM} ) -> ( ${MYDR} )
install -w ;
except ${MYADM}/logfiles;


Rgds...Geoff

[A. Daniel King_1]

Hmmm. Can rdist be told to use ssh?

[Geoff Wild]

Looks like the version of rdist with HP can't use SSH - needs the -P option.

Found this in a SSH fact:

5.3. Can I use rdist or rsync with ssh?
Yes.

If you use rdist, don't forget to compile the path to ssh into it. Alternatively, you may specify the -P option so rdist uses ssh, and not rsh.

If you use password authentication with rdist 6.1.2 through 6.1.5, you will need to apply the following patch to rdist to make it work:

--- src/rshrcmd.c.orig Tue Jun 11 16:51:21 1996
+++ src/rshrcmd.c Tue Jun 11 16:52:05 1996
@@ -63,7 +63,7 @@
/* child. we use sp[1] to be stdin/stdout, and close
sp[0]. */
(void) close(sp[0]);
- if (dup2(sp[1], 0) < 0 || dup2(0,1) < 0 || dup2(0, 2) < 0) {
+ if (dup2(sp[1], 0) < 0 || dup2(0,1) < 0) {
error("dup2 failed: %s.", SYSERR);
_exit(255);
}
This also applies if you get a "Warning: Denied agent forwarding because the other end has too old version." error (which occurs if your client is 1.2.17 or later, and it connects to an older server).


Rgds...Geoff