Running out of Memory

[dave2204]

We have HP3440 server with 16gb RAM running hp-ux 11.23.
Our developers are trying to test a program which appears to run out of
memory. As a test they have created a short C program which uses malloc to
allocate chunks of memory. This currently stops when 990mb is allocated:

allocated 980001176 bytes (980 Mb)
allocated 990001188 bytes (990 Mb)
line 57: malloc error 12 Not enough space

I changed the following parameters one at a time to their maximum allowed
values to see if it helped:
shmmax
shmmni
shmseg
maxdsiz (+64 bit equivalent)
maxssiz (+64)
maxtsiz (+64)

The only one that made a difference was maxssiz. The memory test program
then failed at 660mb instead of 990mb.

This is the output from some commands:

# ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 4194300
stack(kbytes) 8192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

# swapinfo -t
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
dev 8192000 0 8192000 0% 0 - 0 /dev/vg02/lvol2
dev 8192000 0 8192000 0% 0 - 0 /dev/vg01/lvol2
reserve - 636340 -636340
memory 16775168 2777568 13997600 17%
total 37353472 3413908 33939564 9% - 0 -

# vmstat 5 3
procs memory page
u
r b w avm free re at pi po fr de sr
ind
2 0 0 153022 1474838 0 0 0 0 0 0 0
400
2 0 0 148572 1474778 4 0 0 0 0 0 0
400
2 0 0 148572 1474778 0 0 0 0 0 0 0
380

Do you have any suggestions how we can allocate more memory?

Thanks

Dave

[RAC_1 · Accepted solution]

Is the program 32 bit program. They have limitations to memory usage.

[Steve Lewis]

If it managed to allocate LESS memory after you INCREASED parameters then it may be fragmentation that is the problem.
So go for a reboot and re-run the memory allocation test immediately afterwards.
With a rp3440 with 16Gb of RAM, you need to ensure that the program was compiled +DA2.0 and not +DAportable or anything 32bitty.

Then see this excellent thread:
http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=1063391

[dave2204]

Hi RAC

The C program used for testing appears to be 32 bit. Changing maxssiz had
an effect on it whereas chnging maxssiz_64bit didn't make any difference

[Don Morris_1]

Here's the deal.

On HP-UX, the default layout for 32-bit processes is:

0 to 1 Gb -- Text
1Gb to 2Gb -- Static Data, Heap, BSS, Stack and all other private data objects (private mmaps, pthread stacks, etc.)
2Gb to 4Gb -- Shared data [where shared Memory Mapped I/O is place varies by architecture and release, but it is in there somehwere]

malloc() either uses brk/sbrk to increase the heap or uses private MAP_ANONYMOUS mmaps to grow the process private data space [it is up to libc which].

As such -- it should be readily apparent why your mallocs for a 32-bit process begin to fail as you approach 1Gb. Exactly where they fail depends on what _else_ is in the Private space of the process. By increasing the maxssiz tunable (which is the amount of Private virtual address space to reserve for the stack) you decreased the amount of that 1Gb usable for data... hence the malloc failed earlier. Natural consequence.

So if you want more malloc space -- first thing is lower maxssiz back where it was (in general *never* raise maxssiz arbitrarily... most programs need heap more than stack -- if they need more stack [like Java] they'll either tell you when they install or they'll be killed with explicit messages about Stack Growth failure. Only then should you consider raising maxssiz... and if this is a home grown (and not Fortran or Java) program, you should ask the programmer just what kind of recursion games they're playing and why they need such large stacks...)

The best thing to do -- compile 64-bits and get out of the limitations. 64-bit programs on PA work with four quadrants of 4Tb of virtual address space each, with Data/Private being in one of those 4Tb quadrants. You're going to exhaust physical memory or system virtual memory (total swap for reservation) way before you hit that limit.

If you can't do that, you'll need to look into compilation options / chatr options. On PA, there's a compilation option (I think -N, but I may be wrong... check the man pages) to share the first quadrant (0 to 1Gb) as text and data, giving you something a little less than another Gb to play with. After that, you can use options to set q3 private and/or q4 private for more space. Keep in mind that doing so will make using shared objects between other programs more difficult -- 64bit is going to be a lot simpler.

On IPF you also have the option of compiling in the Mostly Private Address Space model (MPAS). This gives you pretty much the whole 4Gb for the program for whatever it wants to put there (text, data, stack, etc.). Shared memory objects are allowed to be mapped in, but they get a private alias in this process.

See the compiler / linker / chatr documentation for more details.

[dave2204]

Many thanks to RAC, Steve and Don.

Problem solved so Thanks very much for your help. Once I concentrated on the 32/64bit issue I was able to compile the program as 64bit and successfully allocated more
memory. So you needn't look any further.