Weird memory problem

[Peter Hug]

I frequently use GlancePlus C.03.72.00 to monitor memory usage un an HP-UX 11i machine.

There are two memory values glance reports, RSS and VSS. What I find under HP-UX is that even though my application deallocates memory, this fact is not reported by glance.

Attached is a small C++ app eatram which can allocate blocks of memory (in multiples of 1MB) from the local heap. If I run this program on an HP-UX 11i and allocate say 500MB of RAM, glance correctly shows RSS and VSS as 500MB. If I then release RAM to 0MB without shutting down the utility, glance reports something like RSS/VSS=300MB/500MB.

While I think Glance may not correctly report eatram's memory usage, what really puzzles me is that when I exit eatram, glance reports that available memory on the machine has grown by the amount of memory it still thought was allocated to eatram.

As far as I can see the C++ code releases memory correctly. Am I experience a problem with Glance or is there something wrong with my code?

Thanks for any feedback in advance
Pete

[Steven E. Protter]

Shalom Pete,

Glance is up to version 4.5. You might want to updgrade and try a different version.

You could also use tusc and hang a tusc trace on the appliction to insure its really releasing the memory. Memory leaks are easy to cause and hard to find and fix.

Always good to run a tool check when you get strange results.

:-)

SEP

[A. Clay Stephenson · Accepted solution]

This is normal behavior whether it is done using the free() function or embedded within a C++ destructor, the memory is not returned to the system but rather is placed back on the process's heap for possible subsequent reuse by the current process. It is possible to actually return memory to the OS via the sbrk() system call using a negative increase parameter but sbrk() can only be used if ALL memory allocation is done via sbrk(). This means that new(), malloc(), calloc(), ... can never be used -- including library functions which might make use of the standard memory allocation tools.

[Peter Hug]

Hi Clay,

Does this mean that if another process needs more memory the OS can reduce the size of my processes heap automatically?

Pete

[Bill Hassell]

> Does this mean that if another process needs more memory the OS can reduce the size of my processes heap automatically?

Nope. The memory remains reserved until the program terminates. Heap or local memory as well as shared memory is never adjusted by the OS. However, HP-UX is a virtual memory system which means that 5 programs that malloc 500megs each will run in 250megs of RAM as long as you have plenty of swap space. (the programs will swap in and out mercilessly)

[Peter Hug]

The main reason I wrote the eatram utility was because I felt that glance incorrectly reported memory usage of my daemon.

What I tyically experience is something along these lines:

Glance reports that my daemon is using RSS/VSS 160MB/220MB. So while my daemon is idle I run eatram and allocate loads of memory from the local heap. Surprisingly, glance shows that my dameons memory usage decreases to 38MB/220MB.

The reduced figure typically coincides with what I experience when my daemon is idle under windows (as a service).

So it appears that somehow that RSS figure can come down if another (hungry) process requires more memory.

How could that be explained if what you stated Bill is correct?

[Steven E. Protter]

Shalom again,

The behavior of your program with respect to a smaller heap depends totally on how it was coded.

Memory can be returned to the heap without releasing it to the OS or changing the heap size.

SEP

[Peter Hug]

I'm simply using the global new and delete operators which I would imagine are implemented in libstdc++.

[Dennis Handly]

>Peter: So it appears that somehow that RSS figure can come down if another (hungry) process requires more memory.

This is correct. RSS measures how much of the process is in memory. (R is for resident.)

>Steven: Memory leaks are easy to cause and hard to find and fix.

gdb has some commands to do leak detection:
info leaks
info heap
You can download gdb from:
http://www.hp.com/go/wdb

>Clay: It is possible to actually return memory to the OS via the sbrk() system call using a negative increase parameter but sbrk() can only be used if ALL memory allocation is done via sbrk(). This means that new(), malloc(), calloc(), ... can never be used --

This is not quite true. The 11.23 mallopt(3) has M_REL_LAST_FBLK so that a free may release the last block with sbrk.

[Peter Hug]

I'm just really lost here...

My daemon runs threads which are activated by some external triggers. Each thread is quite memory hungry. However, I can limit the number of concurrent threads of execution and currently I have set these to 3.

Under Windows and Linux RH I can see that reported memory usage increases/decreases according to the number of threads running.

Also, I have used leak detectors under Windows and there are no problems whatsoever. I haven't tried wdb because I compile my code with gcc and therefore use the gnu gdb debugger which hasn't got these features. Will my code be compatible with wdb?

Somehow, under HP-UX the RSS memory seems to constantly increase. This seems to be consistent with the eatram program I attached to the first message in this thread.

What really concerns me is that glance shows both Mem and Swap as really high and it seems to simply not release memory when I think it should.