ld reports "Malloc is out of memory"

[Michael Kelly_5]

We are getting this error when trying to statically link a (very) large executable. I am pretty certain that this is because ld is trying to malloc more memory than will fit in the data segment.
The kernel parameter MAXDSIZ is set to 1GB.
I also notice that the ld executable is marked as SHARE_MAGIC which limits the data segment to ~1GB anyway.
I'm also pretty sure that there isn't a stray 'ulimit -d' anywhere.

Do I have any options here (aside from not building static executables)?

HPUX 11i with Dec. 2002 Support Plus patches installed.

Thanks.

[Umapathy S]

Check this thread out. This may help you.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc305595fae8bd711abdc0090277a778c,00.html

HTH
Umapathy

[susan gregory_1]

That thread talks about EXEC_MAGIC which will give you up to 1.9~ gigabytes.
Have you looked into SHMEM_MAGIC which can give you up to 2.75~ gigabytes heap?
If you have an EXEC_MAGIC executable, you can then use chatr to modify it into a SHMEM_MAGIC executable.
Of course, you can always go to 64-bit applications which would expand that limit quite a bit (as you can see from the maximum value for maxdsiz_64bit (4,396,972,769,279).

[Mike Stroyan · Accepted solution]

The "chatr +q3p enable" can be done to /usr/ccs/bin/ld32 if you have write permission on the file. You would need to increase maxdsiz as well.
Since you can't relink the linker, EXEC_MAGIC is not an option.
The SHMEM_MAGIC mode does not increase available heap. It steals heap addresses to allow another 1GB of shared memory addresses.