RSS/VSS value increases after excution of "new" but never comes down

[Dibendusah]

Hi All,

We are having concern in the glance report being rendered in HU_UX which depicts the value of RSS/VSS . Basically we are doing dynamic memory allocation in our application. Or in other words using "new" and "delete" of c++ . Whenever we are allocating memory using "new" , we could see increased value reflecting right away in that RSS/VSS. However the shown value is not coming down after the executing of "delete" even after waiting for several minutes ( i.e. doing other works in that programme without exiting after executing of the “delete” ) . This is actually confusing because, we expect if RSS/VSS value is increasing upon execution of "new", the same value should be seen going down upon execution of "delete" .

code is pretty straightforward and shown below:

=============================

#include<iostream.h>

#include<stream.h>

int main()

{

       int *temp;

       cout<<"Before the action"<<endl;

       sleep(120);

       int n;

       for(int i=1;i<5;i++)

       {

               cout<<"\n Iteration :\t "<<i<<endl;

                n= 10737411;

               temp = new int[10737411+n*i];

               cout<<"memory acquired"<<endl;

               sleep(30);

               delete[] temp;

               cout<<"de-allocation done"<<endl;

               sleep(30);

       }

       sleep(10);

return 0;

}

=============================

Corresponding observation through glance:

Type             RefCt     RSS     VSS          Locked File Name

DATA /Priv     1       92kb       96kb     0kb   /u02/.../test_cc.exe ( start of test programme)

DATA /Priv     1      140kb    163.9mb               0kb   /u02/.../test_cc.exe   ( after allocation )

DATA /Priv     1      140kb    163.9mb     0kb   /u02/.../test_cc.exe   ( after de-allocation of memory acquired)

DATA /Priv     1      156kb    204.9mb     0kb   /u02/.../test_cc.exe   ( allocation of bigger chunk )

DATA /Priv     1      156kb    204.9mb     0kb   /u02/.../test_cc.exe   (de-allocation of bigger chunk)

DATA /Priv     1      156kb    204.9mb     0kb   /u02/.../test_cc.exe   (Process stopped)

What is the meaning of this behavior ? Whether "system memory allocation" algorithm implemented by HP_UX works little differently than we are expecting ? Whether memory acquired will not be released back to OS ? if yes then when and how we will be able to see it. We are seeing it never coming down ( RSS/VSS value ). How to determine , at any given point in time , the actual memory (resident memory) acquired by any process in HP_UX. Since the value appearing in RSS never comes down, does that mean: the acquired memory is not available to other process as long as the first process is running ? Or it will be available to others ( if so how to identify that ). We are using HP_UX Itanium version 11.31

Please clarify . Appreciate your time and valuable inputs.

Thanks in advance.

Thanks & Regards

Dibendu

P.S. This thread has been moved from HP-UX>General to  HP-UX > languages. -HP Forum Moderator

[Dennis Handly]

>Whether memory acquired will not be released back to OS?

This is how the heap works on many OSes.  Once acquired it is kept around, assuming it eventually be reused.

>Since the value appearing in RSS never comes down

Your RSS doesn't come down since it only increased by a trivial amount since you didn't touch all those pages.

The RSS should come down if you are allocating multi-page blocks, when there is memory pressure.

>does that mean: the acquired memory is not available to other process as long as the first process is running?

It depends, the pages could be swapped out and then reused by other processes.

If you are allocating huge blocks for a short period of time that you want to go away, you could look at mmap(2) and munmap(2).  For C++, you can write your own allocators.

These previous topics may help:

http://h30499.www3.hp.com/t5/Languages-and-Scripting/Vector-doesn-t-release-memory/m-p/4095208

http://h30499.www3.hp.com/t5/Languages-and-Scripting/vector-STL-doesn-t-release-memory/m-p/4095214

http://h30499.www3.hp.com/t5/Languages-and-Scripting/How-to-find-released-memory-of-a-process/m-p/4481644

[Dibendusah]

Hi Denis and All,

Thanks a million for your quick and valuable inputs. It really helps.

In order to get the exact instantaneous free and used block , we used GDB provided "info heap processes" command. The total memory allocated for ARENA is divided as below

Total space in arenas

Number of bytes in free small blocks

Number of bytes in used small blocks

Number of bytes in free ordinary blocks

Number of bytes in used ordinary blocks

Number of bytes in holding block header

We are considering free blocks as : Number of bytes in free small blocks + Number of bytes in free ordinary blocks +

Used blocks as : Number of bytes in used small blocks + Number of bytes in used ordinary blocks + Number of bytes in holding block header

Following is a small extract of the data gathered out of it.

When process just started , before memory allocation :

Glance:

RSS/VSS                                              92 kb/96 kb     

GDB Info heap process data &colon;

Total Free in blocks                         0.04776001 mb

Total Used in blocks                      0.02976989 mb

After Memory allocation :

Glance:

RSS/VSS                                                           108kb/82.0mb

GDB Info heap process data &colon;

Total Free in blocks                                        0.045379639 mb

Total Used in blocks                                      81.95011902 mb

After memory de-allocation :

Glance:

RSS/VSS                                             108kb/82.0mb

GDB Info heap process data &colon;

Total Free in blocks                         81.96533203 mb

Total Used in blocks                        0.030166626 mb

So We are seeing that even though RSS/VSS value does not come down but :

The value of VSS = Free blocks + Used blocks.

1> When memory allocated : free was 0.04mb and Used was 81.9mb

2> When memory de-allocated : free was 81.9mb and Used was 0.03mb

Is our above understanding is correct ?

Can we conclude that after de-allocation the "free blocks" are available for further use(same process/other process) ?

is there way we can find out something like (Free + used) of RSS ? We are not sure how to calculate RSS from GDB provided statistics

I am attaching one excel with more clean and detailed statistics collected from both glance and GDB for each allocation and de-allocation iteration. Please refer it for more elaborate data.

Thanking again for having a look and sharing your thoughts and experience.

Thanks & Regards

Dibendu

[Dennis Handly]

>we used GDB provided "info heap processes" command.

You can look at mallinfo(3).  But it doesn't work too well for a 64 bit process.

>Is our above understanding is correct?

Yes.

>Can we conclude that after de-allocation the "free blocks" are available for further use (same process/other process)?

Only by same process.

>is there way we can find out something like (Free + used) of RSS?

No.  RSS is related to when the page was touched last, not if it is free.  In your test case RSS is approximately 0.

>We are not sure how to calculate RSS from GDB provided statistics

Only the kernel knows how to compute RSS.  But you might be able to use pstat_getprocvm(2).

>I am attaching one excel with more clean and detailed statistics collected from both glance and GDB

Is this for your real application?

[Dibendusah]

Hi Dennis,

Thanks again for clarifying our doubts.

The data I attached is from teh sameple program whcih I have pasted in this thread . Basically our real application behaviour  is inline with this.

You mentioned that :

after de-allocation the "free blocks" are available for further use by the same process only. This is little confusing. Because in your first response, you mentioned that under pressure situation the free mememory can be used by other processes. OS should be taking care of required action for this purpose.

Appreciate your inputs this direction.

Thanks & Regards

Dibendu

[Dennis Handly]

>you mentioned that under pressure situation the free memory can be used by other processes.

Sorry for the confusion.  Pages on the free list are still part of the VSS and can be swapped out of memory and the RSS reduced.  Then the physical pages can be reused by another process.

So there are virtual pages that can't be reused and physical pages that can.  But you must have enough swap to hold that VSS.

(Virtual pages can also be shared with mmap(2) and shared memory.)