freeing up memory

[Rhaul]

On HPUX-11.31, a program allocates memory upto 1000M and then then starts de-allocating all the memory. The program keeps running after de allocation of the consumed memory.

However, the process size does not decrease (from 1000M) for hours. Could anyone please help me understand this memory management behavior? When will this 'freed up' memory be actually made available for reuse by this process/other processes?

[Don Morris_1 · Accepted solution]

That's normal (and not atypical for most OS's).

The thing to understand is that the allocation is simply for virtual space. (If the allocated memory is touched, then physical pages could be associated, but they may not). The free() operation returns the virtual ranges as being available to the data area in the library space, but the library is free to cache those virtual ranges for later reuse. This is all assuming the allocation is via malloc() [in C] or the equivalent. If the library doesn't ever call the kernel to shrink the data area [the brk()/sbrk() system calls], the library and process both still have all this virtual space.

As to why this doesn't matter that much in most cases -- as mentioned above, there may not be physical space affiliated with these ranges in the first place, and if there is but no one is using it and the system comes under memory pressure, the kernel is free to take those physical pages and give them to other processes (since these will not be currently accessed, it makes them more likely to be paged to swap and the physical memory freed for re-allocation). On the virtual side - these virtual addresses are private to the process in the first place, so they're not preventing other processes from doing allocations in their private space [other than reserving swap resources which might be needed in the worst case].

There are a couple of scenarios I can think of where it would be nice to actually shrink the heap down (low swap reservation space, wanting to do additional private mmaps and having the heap stuck in the way). There may be a way to tell libc to release this stuff more quickly -- I don't know of one off hand [Dennis? Anything like that you know of?]. From the kernel perspective, there's no way to know what is really still in use versus cached by the library level, so it would have to be a library functionality.

[Rhaul]

Thanks very much Don.
The overall problem here is that the process consuming 1000m is a 32-bit binary. There is a limit of 1g process size for 32-bit processes. So after this program is de-allocating all of the memory used, any subsequent invocations of the program fail to even allocate small amount of memory like 5M - resulting in malloc failures.

The process size is still listed as 1g in top/glance.
Any suggestions how to overcome this -such that subsequent invocation of the program (after a memory de-allocation) have the de-allocated memory available for re-use?

[Ishwar_1]

Can you post the output for the :-

1> swapinfo -tam
2> Check what is the Load on the System : top
3> which are the One Consuming the Resources :-
System, Physical, USers, Swap, sleep, running process

[Dennis Handly]

>There is a limit of 1g process size for 32-bit processes.

If you work hard, you can get ~4Gb. It's pretty easy on Integrity.

>after this program is de-allocating all of the memory used, any subsequent invocations of the program fail to even allocate small amount of memory like 5M - resulting in malloc failures.

You have NOT freed ALL of memory.
That could only happen if you have fragmented the heap or have freed every other block so there is no 5 Mb region.

>such that subsequent invocation of the program (after a memory de-allocation) have the de-allocated memory available for re-use?

How do you know how much memory is freed? Have you used mallinfo(3)?
Are you using threads with many arenas?

>Don: Anything like that you know of?

There is mallopt(M_REL_LAST_FBLK) but it may not were as well as you would like.

[Don Morris_1]

I think you may want to clarify what you mean by "subsequent invocations" of the program (or where this allocation is). Because other than swap reservation consumption as mentioned, it doesn't matter how many different 32-bit 1Gb eating systems you have running, each of those 1Gb allocations are private to the process.

Only if this is _shared_ address space could you possibly starve another invocation of the process for address space. And giving back shared address space (unlike a free() to the heap corresponding to a prior malloc()) _is_ immediate once the last reference releases it. (You can't give it back while folks are still using it after all).

So either you mean something other than launching a new process by "invocation", you're talking about something other than malloc and the heap (and most of what we've said isn't relevant) -- or we have a misunderstanding on what impact process private allocations have on new processes.