Memory usage /shared memory

[Roger Baptiste]

Hi,

Where does measureware/perfview show the "shared memory"consumption i.e the memory as shown by ipcs -am command??
This question arises, because i am trying to do a break up of the GBL_MEM_UTIL consumption at the application level. It is obvious that
GBL_MEM_UTIL is = GBL_MEM_USER_UTIL + GBL_MEM_SYS_AND_CACHE_UTIL.
Then, drilling down the GBL_MEM_USER_UTIL further, i look up at the APP_RES_MEM consumed by each application (as defined by parm file) and sum it up. But the sum of all the application's APP_RES_MEM at a given time does not equal GBL_MEM_USER_UTIL. The difference in the values seems to almost equal the shared memory shown by ipcs -am.
Is this presumption correct??
If so,
Is there a way to have Measureware capture this data too?

thanks!
-raj

[Sridhar Bhaskarla]

Raj,

I don't have an answer. But there is a table tbl_shmem_requested available with glance though to get the current shared memory usage (with few exceptions). If you want to get the data, you can try it by keeping the line "print tbl_shmem_requested" in a file and running glance over it non-interactively (glance -j 2 -i 2 -syntax file -adviser_only)

Regarding your presumption, the definition of APP_MEM_RES as in /opt/perf/paperdocs/mwa/C does say that it includes shared memory regions.

I am not sure why you are seeing the difference equal to shared memory usage when you add all the application's APP_MEM_RES. Probably, as the shared memory part reported is the process' portion of the shared memory, depending on your application definition, you may not be getting the actual value of the shared memory used. Did you try adding APP_MEM_RES values for all the applications including "Other"?

-Sri

[Martin Johnson]

You should be able to get this from Measureware. The best way is probably with /opt/perf/bin/utility and export a report. If you are not familiar with this utility, use the guide function to help you generate a report.

HTH
Marty

[John Poff]

Hi Raj,

Take a look at these:

APP_MEM_RES
APP_MEM_UTIL

GBL_MEM_USER
GBL_MEM_USER_UTIL

PROC_MEM_RES
PROC_MEM_SHARED_RES
THREAD_MEM_SHARED_RES
PROC_MEM_VIRT

PROC_REGION_RES_SHMEM
PROC_REGION_VIRT_SHMEM

TBL_SHMEM_AVAIL


According to the MeasureWare dictionary docs, these include shared memory. I tried the last one on an 11.00 box here that is running an old version of MeasureWare (c.2.30) and it seemed to work.

I went to this URL:

http://ovweb.external.hp.com/lpe/doc_serv/

and looked in the dictionary for one of the latest versions of MeasureWare (agent for Unix) and found these metrics.


JP

[Mladen Despic]

Raj,

For the sake of simplicity, let's assume that your app, as defined in your /var/opt/perf/parm file, consists of 4 processes: p1, p2, p3, p4. Let's assume that these four processes are attached to the same shared memory segment, of size, say, 800Mb. Furthermore, let's assume that p1, p2, p3, p4 have no other "shared RSS" and the "private RSS" is about 100Mb each. Obviously we want the metric APP_MEM_RES to represent the total RSS of the four processes as follows:

APP_MEM_RES = 4 x 100Mb + 800Mb = 1200Mb

However, Glance and Measureware evaluate APP_MEM_RES simply by adding PROC_MEM_RES for each of the four processes. So, if you say that

PROC_MEM_RES = 100Mb + 800Mb = 900Mb,

which you may consider to be a true statement, then your APP_MEM_RES would be equal to 4 x 900Mb = 3600Mb. This result is obviously incorrect.

I think this problem was present in one of the earlier HP-UX versions (10.X ??). Under HP-UX 11.00, however, Glance and Measureware evaluate PROC_MEM_RES differently. I believe the new formula is:

PROC_MEM_RES = sum of private region pages +
(sum of shared region pages /
number of references)

So, in the example given above, we would have:

PROC_MEM_RES = 100Mb + ( 800Mb / 4 ) = 300Mb

Now the total of all PROC_MEM_RES metrics would correctly evaluate to:

APP_MEM_RES = 4 x 300Mb = 1200 Mb

For processes that are attached to a shared memory segment only once, this should work without a problem. However, the "number of references" is sometimes incorrectly reported from the kernel. HP says this bug should be fixed under HP-UX 11.11.

I haven't used 11.11 yet, but I would expect that the total of all APP_MEM_RES is now a lot closer to GBL_USER_MEM. However, if I hear HP correctly, the metrics APP_MEM_RES and GBL_MEM_USER should still be only approximations.

If you want to see the relevant "reference numbers" and MEM_RES types of metrics on your system, try the following:

1. Save the attached file, let's call it shmem.syntax, to your current directory.

2. Run:

glance -adviser_only -syntax shmem.syntax -iterations 2 2>/dev/null

3. Compare the shared memory segments from the glance output with the output from 'ipcs -mopb'.

4. The output from 'ipcs -mopb' should show the number of attached processes to each shared memory segment under the columns NATTACH. Can you confirm that this number is reflected in your Glance output? (This is where you might observe the discrepancy.)

HTH,

Mlade

[Mladen Despic]

Raj,

To clarify the very last point of my previous post,
I've attached an example from my system. The output from 'ipcs -mopb' shows a 1-Gb shared memory segment belonging to 'sybase'. The Glance output shows only 4 processes being attached to this shared memory segment, whereas NATTCH column gives 18 attachments.

To evaluate PROC_MEM_RSS for each 'dataserver' process in this example, the 1-Gb shared memory segment should be divided by 4, as this is the true number of attached processes. But Glance divides it by 18 instead, as this is the number attachments as received from the kernel. Thus the actual PROC_MEM_RSS is lowered by about 200 Mb for each 'dataserver' process.

Mladen

[Mladen Despic]

I meant to say "Glance divides it by 12 instead ..."

[Roger Baptiste]

Hi Mladen,

Thanks for taking the time to explain this in detail. My comments are below:

<< I think this problem was present in one of the earlier HP-UX versions (10.X ??). Under HP-UX 11.00, however, Glance and Measureware evaluate PROC_MEM_RES differently>>

Yes, the docs mentioned that problem was in 10.20 and the PROC_MEM_RES is calculated more accurately in 11.00, as per the new formula.
In my case, i do use 11.00, so i would expect a closer approximation between the Sum of the APP_RES_MEM and GBL_MEM_USER_UTIL.

<>

Any idea, how big a difference does this bug result in?? For eg: i find the difference between GBL_MEM_USER_UTIL and the sum of APP_RES_MEM of all applications (including "others" as shown in perfview) to be around 2GB. That seems too large and also the bigger number is the GBL_MEM_USER_UTIL.
eg:
GBL_MEM_USER_UTIL ~ 50% on 14GB memory system
~ 6.5 GB
SUM of all applications APP_RES_MEM ~ 4.5GB
That leaves a gap of unaccounted 2Gb.

Is it a coincidence that the sum of all shmem values as shown in ipcs output is almost equal to 2Gb!! ??

<>

Thanks for a useful syntax script. I ran it and it provided intresting output.
For instance for oracle application, there are two shared memory regions and the NATTACH for each of this is 2836 and 1890 respectively, as shown below:

---------------------
m 11282 0x84407e34 --rw-r----- oracle dba 2836 997928960 14133 23380
m 5142 0x00000000 --rw-r----- oracle dba 1890 1328545792 14133 23380
------------------

Whereas in the output of the glance script, the total number of oracle processes being shown with these shared memory attachments is 943. A sample extract of the glance output is:
---------------------------------------------------------------------------
ora_p004_db 14865 34.0mb
1.24gb 1.24gb
951.7mb 951.7mb
---------------------------------------------------------------------------

From the above output, can the PROC_MEM_RES be taken as = 34mb+ (1.2gb+951.7mb)/943 or
= 34mb + (1.2gb/2836 +951.7mb/1890)?
(or is this badly off the mark! ;-) )

Also, as seen above a typical oracle process is using both the shared memory segments and there are 943 such processes in the output of the glance script. Can this be considered a discrepency -The NATTCH being more than the ones being shown by glance? Even if it is so, would there be such a huge 2Gb difference?

Where exactly does it get the "number of references" from??

And how to explain the missing memory being almost equal to the ipcs shared memory output?
I checked on another system and the "discrepancy" seems to exist there too.

Any thoughts? Thanks again!
-raj

[Roger Baptiste]

Mladen,

Your sybase example seems to coorelate with what i see on my system. ie the NATTCH on the oracle shared mem being much higher compared to the glance output of the script you gave.
NATTACH = 2836 and 1890 for the two shared memories of oracle.
Whereas glance output has 942 processes.
The difference is almost 3 times (just lke in your case, 4 ~ 12 ).

So, is this the actual problem? and is it a known bug? and is this causing the difference between the sum of app mem and the glb user mem??

Thanks again! That was quite neat!
-raj