Heap fragementation on HPUX

[Atul Goel]

The Resident size(as observed from top) of my process is increasing. But, the behaviour is very random. My process works on request reponse model. So when i put some request load on my process the memory starts increasing. For initial few hours (approx ~3 hrs) it increase at a rapid rate and after that increase become stable. Some time after 30 min a 16 kb increase is observed some time after 2-3 hrs. And some times an increase of even 128kb is observed. Behaviour is really random. Attached are the some graph of resident memory of process taken at interval of 10 min each.

I tried wdb in BATCH mode with my process. But, it didnot detected any leaks. It produced only heap profile files.

Is there any way to find out that the wt=hether increase is due to fragmentation? Because lot of malloc are happening in my process.
Platform is HP-UX B.11.23 on HP itanium server. I can use wdb and glance tool(I only have working knowledge of both of these :( )

Also, considering the fact that memory increase is after 2-3 hr run how can i work with wdb in debugger mode. Means is it possible to automate printing the heap information and continuing from that point in gdb/wdb automatically?

[James R. Ferguson]

Hi:

The heap (from 'malloc' and cousins) may grow, but it will not shrink. When 'malloc()'ed memory is 'free()'ed that memory is returned to the heap, but not to the system until the process terminates.

Regards!

...JRF...

[Steven E. Protter]

Shalom,

Did you try my handy dandy memory leak detector?

http://www.hpux.ws/?p=8

SEP

[Atul Goel]

Hi James,
Thanks for quick reply. But, how to ensure that there is no memory leak. Because, if i monitor process for say 7 hr or 8 hrs i can see some jumps of memory increase in 16kb chunks.

I have to prove that there is no memory leak but i cannot integrate with any tool like purify which need recompilation due to some techinal difficaulties

[Atul Goel]

Hi Steven

Your tool will tell which process is taking memory. I know there is an increase in memory of my process but i am not sure wether it is because of memory leak or something else

[Dennis Handly]

>The Resident size (as observed from top) of my process is increasing.

(You don't care about Resident size, just the total size but I assume both are increasing.)

>it did not detected any leaks. It produced only heap profile files. ... Is there any way to find out that the whether increase is due to fragmentation?

Then you either have heap fragmentation or your heap usage is increasing. Did wdb say whether your heap allocation was increasing or just the arena and free areas? The latter is heap fragmentation. (Are you using threads?)

Fragmentation can also be detected by mallinfo(3).

>I only have working knowledge of both of these

Have you looked at the WDB document: Debugging dynamic memory usage errors using HP WDB
http://www.hp.com/go/wdb

>is it possible to automate printing the heap information and continuing from that point in gdb/wdb automatically?

You can also call mallinfo(3).

>I have to prove that there is no memory leak but I cannot integrate with any tool like purify which need recompilation

No need to use purify. wdb/mallinfo(3) will tell you if you have leaks but not where.
Also, whether you have a leak or heap fragmentation, the result is the same, except instead of blaming your application, you blame libc. :-) Or you need to change how you allocate space

You may want to use MallocNextGen:
http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=MallocNextGen

It does a better job if you have threads.


>JRF: The heap may grow, but it will not shrink.

If Atul mentions "Heap fragmentation" he probably knows about this. He needs to look for zebras. ;-)

[Laurent Menase]

if it is fragmentation problems you can play with mallopt() to specify how small block allocator should behave.

you can also make tests with _M_SBA_OPTS env variable to test various sba configs.

[Atul Goel]

Hi Dennis

Thanks for the detailed information. But, when you say "Also, whether you have a leak or heap fragmentation, the result is the same" do you mean we cannot distinguish between the two. I hope there should be some way to distinguish between fragmentation and leak and i am looking for the same

[Atul Goel]

Hi Laurent

Can you please more information regarding this "_M_SBA_OPTS" variable. Or can you point to some good links regarding this option

[Laurent Menase]

The only doc I know is man malloc