memory allocation 32bit vs 64bit puzzle

[Roman Guoussev-Donskoi]

32 bit test application cannot allocate even 660M on this machine while the same app compiled 64bit can allocate 800M.

According to kmtune:
maxdsiz 0x80000000 - 0X80000000
maxdsiz_64bit 0x80000000 - 0X80000000
allocate_fs_swapmap 0 - 0
maxswapchunks 4096 - 4096
nswapdev 10 - 10
nswapfs 10 - 10
remote_nfs_swap 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048

ulimit -Sa
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 392192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 512

cc -o memhog32 memhog.c
cc +DA2.0W -o memhog64 memhog.c

./memhog64 900000 1 20
total space 921600000
Allocated 0 chunk.
received signal

./memhog32 660000 1 20
total space 675840000
Allocation failed. Null pointer.: Not enough space

Note: I have executed this test repeatedly with the same result always.


./meminfo
Memory Stat total used avail %used
physical 2048.0 1216.8 831.2 59%
active virtual 2207.7 1753.9 453.9 79%
active real 455.5 350.8 104.7 77%
memory swap 1530.0 501.0 1029.0 33%
device swap 2048.0 1556.0 492.0 76%

On another machine 32 bit version works happily up to almost 1024M (as expected).

[A. Clay Stephenson]

Your result is not too surprising in 32-bit land. In fact, I never count on more than about 512MB in 32-bit land because I plan for the worst-case. About as good as you are ever going to do is about 960MB and that is with a lightly loaded system with a "clean" heap. As the heap becomes more and more fragmented it becomes more difficult to find a single big enough chunk of memory to grant your request. I suspect that if you reboot (and start your program before anything else (e.g. oracle) is loaded) you will be able to allocate a large enough chunk. Note that you are really using a large fraction of your swap so that you may need additional swapspace. You should also make certain that your buffer cache settings are reduced to no more than about 10% (or statically set by setting a non-zero bufpages).

[Roman Guoussev-Donskoi]

Thank you for your reply.
Could you please clarify some points?

Why on the same machine 64 bit application allows to allocate more memory?

Since both allocations are significantly less than 1G (1 quadrant size) how come 32 bit application nature come into equasion?

Aren't 64 bit applications affected by fragmentation in the same way as 32 bit apps?

You mention 512M (worst case).
Would you have any pointers where I can read more about fragmentation and about this worst case?

Thank you.

[A. Clay Stephenson]

The difference is that in 64-bit land there are essentially no limits (other than maxdsiz_64bit and available virtual address space). In 32-bit land you quickly hit the 1GB quadrant limits (less any reserved space). Worst-case can be considerably worse than 512MB; that's just my personal worst worst-case. If I think I'll need to malloc/calloc/realloc more than about 512MB in 32-bit land, I typically re-think the problem and start looking at other approaches like shared memory. Of course, unless you must link with libraries for which only 32-bit versions exist, the smart play is to compile as 64-bit code so that none of these limits apply.

Even in the case where you must use 32-bit libraries, a good approach is to split your task so that it is mainly 64-bit and fork() and exec() a 32-bit child that communicates with the parent over IPC or pipes.

[Roman Guoussev-Donskoi]

I understand general approach. Unfortunately I'm faced with 32bit packaged application which I cannot change that seems to start failing on memory allocation well before hitting 1G limit.

I have reviewed kernel params listed in my initial message and did not find anything.

Therefore I have created a small test application to simulate memory allocation.

At this moment I do not have a choice of changing the original packaged application but I'm requested to quantify/explain why this (32 bit) application cannot allocate even 700M total while on the same system 64 bit application can allocate 800M. Since both requests are way below 1G limit it is not at play here since requested memory is way lower.

Are there any other factors that account for difference of maximum memory that 32 and 64 bit applications can allocate (assume total requested space is under 800M)?

Thank you.

[Don Morris_1 · Accepted solution]

Keep in mind that maxssiz (which judging from your stack ulimit you've set to about 383Mb) sets aside space in the private area of 32-bit processes (usually 1Gb) to allow the stack to grow to that limit. If you want a larger data allocation, you're going to have to lower your maximum stack size.

[Roman Guoussev-Donskoi]

Thank you, Don!
You have hit it right in the head!
I'd give you 20 points if I could.