topic Re: Malloc (HP's memory handling) in Operating System - HP-UX

Malloc (HP's memory handling)

marius etsebeth_1 — Mon, 08 Sep 2008 10:27:32 GMT

Hi everyone,

I hope someone can help. Currently we have a Unibase application running out of memory every so often. When it is initially started, the memory usage is fine, but this builds up gradually over time until eventually the machine is out of memory. The APP is then stopped and restarted, memory grows etc..

Now, to me, this looks like a memory leak somewhere. The Unibase developers are adamant that this is an HPUX issue (see their comments attached).

Can anyone shed some more light before they convince us to buy more memory, which may ultimately be a waste?

Thanks!

Re: Malloc (HP's memory handling)

James R. Ferguson — Mon, 08 Sep 2008 10:55:57 GMT

Hi:

A 'malloc()' allocates memory in the heap. A subsequent 'free()' returns that memory for reuse to the heap, but not to the system at-large. Upon termination of a process, the entire heap is returned to the system.

Regards!

...JRF...

Re: Malloc (HP's memory handling)

Steven E. Protter — Mon, 08 Sep 2008 10:55:58 GMT

Shalom

http://www.hpux.ws/?p=8

Memory leak detector.

The response to a memory leak is to FIX it, not buy memory.

Once you show unibase evidence of a memory leak associated with one of their processes they'll have to fix it.

SEP

Re: Malloc (HP's memory handling)

Dennis Handly — Mon, 08 Sep 2008 11:03:54 GMT

As JRF and Unibase says, the heap will grow and never return memory to the system.
So if your application uses lots of memory in the middle and frees most of it near the end, that application will need to exit.
Or be reprogrammed to use shared memory or mapped files.

Or you deal with this by adding lots of swap.
Do "swapinfo -tam" when you said "the machine is out of memory".

>The OS-process resources will not release (or decrease in size) after the processing completes

This isn't quite true. The VM use won't decrease but the memory should/could just be paged out.

Of course there still could be a leak in the application, check with wdb.

Re: Malloc (HP's memory handling)

Duncan Edmonstone — Mon, 08 Sep 2008 12:05:35 GMT

The statement from Unibase is broadly correct although it doesn't give much context.

Allocating memory is very expensive in CPU-time (as your UNIBASE person found when he tried using the other malloc library), so its generally preferable to leave the free'd memory in the processes data segment.

There are some fairly esoteric ways of tuning the behaviour of malloc if you're interested (look at the man page for malloc(3c) in the "EXTERNAL INFLUENCES" section), but I would be wary of using these without knowing exactly what you're doing. Better to request some assitance via the response centre for this (although it might be considered "consulting" and mean a cost!)

HTH

Duncan

Re: Malloc (HP's memory handling)

Dennis Handly — Mon, 08 Sep 2008 19:57:08 GMT

There could be N problems here:
1) An ordinary memory leak in the application.
2) Heap fragmentation, looks like a leak.
3) Swap space exhaustion because space is kept until program exit.

Since you said "memory grows", it probably isn't 3) like I first mentioned??

Re: Malloc (HP's memory handling)

James R. Ferguson — Mon, 08 Sep 2008 21:42:09 GMT

Hi (again):

It occurs to me to also ask, do you ever issue 'kill -9' ? The point of the question is that your memory exhaustion could be at least partially a result of leaving shared memory segments lying around.

Regards!

...JRF...

Re: Malloc (HP's memory handling)

marius etsebeth_1 — Wed, 10 Sep 2008 06:54:45 GMT

Hi all,

Thanks for the replies so far. Allow me to answer a few of your questions:

1.) Personally, I haven't used kill -9 on these machines, but it may be that someone else with the necessary access has...

2.) Swap space utilization was yesterday at 31%; today it's 43%. Still nowhere near fully utilized.

Quite frankly, I don't know how to chase a memory leak or even ordinary memory fragmentation in someone else's APP. And unfortunately, the APPS owners aren't too keen on assisting either as they want us to put in more memory.

I'll see what I can come up with in the next day or so and will then allocate points to all accordingly.

Marius

Re: Malloc (HP's memory handling)

Dennis Handly — Wed, 10 Sep 2008 07:02:25 GMT

>I don't know how to chase a memory leak

Look at SEP's link for the simple/big picture.
For WDB:
http://www.hp.com/go/wdb
Go to the documentation link and then download:
Â» Debugging dynamic memory usage errors using HP WDB

>even ordinary memory fragmentation

You probably can't. :-(
Looks like a leak except libc/wdb/purify claims that there is no growth in allocations, just in free blocks.

>in someone else's APP.

You just need to know how to run wdb/gdb.

Re: Malloc (HP's memory handling)

marius etsebeth_1 — Tue, 16 Sep 2008 09:48:19 GMT

Thanks to all who replied.

I have learned one or two things that I may use, but ultimately, it would depend on the co-operation of the Unibase people, especially if I want to go down the debugger -route.

Marius

Re: Malloc (HP's memory handling)

Duncan Edmonstone — Thu, 18 Sep 2008 08:05:05 GMT

marius,

you might also want to point the unibase folks at this new next gen malloc routine in the latest release of 11iv3 (11.31) - might be worth them looking at as well:

http://docs.hp.com/en/5992-4174/ch10s09.html

HTH

Duncan

Re: Malloc (HP's memory handling)

Dennis Handly — Thu, 18 Sep 2008 19:25:18 GMT

>especially if I want to go down the debugger route.

Actually if you know what you are doing, you don't need their help (unless stripped), unless you want them to change things, as Duncan suggests. :-)