Process memory and swap reservation

[Petr Wünsch]

Good afternoon,

I am pretty sure this toppic has been discussed many times here on ITRC, but I am still not able to make a complete picture.

I am tryng to find out how to check memory used by processes on HP-UX (v2 and v3) systems. I know I have several possibilities to do so, but none of then seems to be reliable enough...

1) top
The top command gives me two values to chose from: SIZE and RES - both in KB.

Citing from man page:

SIZE:
Total virtual size of the process in
kilobytes. This includes virtual sizes of
text, data, stack, mmap regions, shared
memory regions and IO mapped regions. This
may also include virtual memory regions
shared with other processes.

RES:
Resident size of the process in kilobytes.
It includes the sizes of all private
regions in the process. The resident size
information is, at best, an approximate
value.

End of citation.

I find SIZE mostly unusable as it shows memory usage including shared segments, which is mostly misleading especially when you run severla oracle database instances on a single system. Simple sum of all proces SIZE values will give you a number much higher then what your physical memory rally is.

For some time I used these steps (found probably here on ITRC) to check process memory usage:
# top -f /tmp/top.out -n 10000 -d 1
# sort -n -k7,7 /tmp/top.out (sort by SIZE)
# sort -n -k8,8 /tmp/top.out (sort by RES)

However, if I list all processes and add up all RES sizes I usually do not get the same or even similar result compared to kmeminfo.

2) ps
It is the same story actually...
A well known ps command (or one of its incarnations) cited many times here on ITRC:
# UNIX95= ps -e -o 'vsz pid ruser args' |sort -nr
gives me also size in KB including shared segments. Thus adding up all processes on my 16GB server tells me i have 130GB memory allocated by processes. Not really helpful.

3) glance - same stroy as top
List of processes gives me two different values for memory usage RSS and VSS:
RSS does not include shared memory, but is inaccurate (I believe it is the same value as RES in top).
VSS shows shared memory.

So, on my server I get these results:
# kmeminfo
Physical memory = 3133186 12.0g 100%
Free memory = 408282 1.6g 13%
User processes = 1909499 7.3g 61% details with -user
System = 802703 3.1g 26%

# swapinfo -tam
root@itan06a2: # swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 679 3417 17% 0 - 1 /dev/vg00/lvol2
dev 2048 0 2048 0% 0 - 2 /dev/vg00/lvol11
dev 2048 0 2048 0% 0 - 2 /dev/vg00/lvol12
dev 2048 0 2048 0% 0 - 2 /dev/vg00/swap3
reserve - 8724 -8724
memory 12239 8655 3584 71%
total 22479 18058 4421 80% - 0 -

- sum of VSZ from ps or SIZE:
130GB

- sum of RES from top:
4,3GB

To sum it up:
- kmeminfo tells me I am using 10,4GB of my 16GB of physical memory
- swapinfo tells me 8655MB of memory is used and that there are 18058MB of swap devices reserved
- ps and topp tell me I am using either 130GB or only 4,3GB

I hope you can see my confusion here...
Either I am terribly mistaken in my assumptions or I need a proper explanation.

Anyhow, what I would really like to be able to do is:
- find processes which consuming most of the memory (misbehaving system process or oracle job?)
- see how much memory and swap (reservation) are individual processes consuming

I would rally appreciate any useful hints on this topic.

Regards

Petr

[P Muralidhar Kini]

Hi Petr,

>> find processes which consuming most of the memory (misbehaving
>> system process or oracle job?)

* Memory Usage by process - Always the BIG ?
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1276204496934+28353475&threadId=1426124

* how to find the process that uses the most memory on hp-ux
http://fixunix.com/hp-ux/373453-how-find-process-uses-most-memory-hp-ux-top-doesnot-do.html

* Memory Utilization on HP- unix 11.23
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1272645736183+28353475&threadId=1224985

Hope this helps.

Regards,
Murali

[Dennis Handly]

If you write a simple program to call pstat_getprocvm(2), you can get all of the details you want. It can tell you if the memory region is shared or not.

[Steven E. Protter]

Shalom,

I think the ps output with the fields displayed is the best way to go.

You have to decide if you are looking at individual processes or overall stats. Some of the overall stats tools are counting the buffer pool, and perhaps other memory areas wrong.

It should also be known that processes can blow off system memory limitations on 11.31 simply by using memory maps.

Back to topic, for individual processes, ps is best IMO, for overall use, I think kmeminfo or swapinfo -tam is sufficient.

Different calculation methods are going to produce different results.

SEP

[Tingli]

For memory usage:

UNIX95= ps -eafo user,time,pid,vsz,args | sort -rk 4

For swap usage:

swapinfo -tam

[Petr Wünsch]

Thank you all for your suggestions.

Murali:
Some of those links proved to be quite useful - provided some more hints or explanations.

Dennis:
That might be an interesting task, I will keep it in my "wish list".

Anyway, after reading many forum posts and testing different tools and approaches I believe there is no simple answer. As expected.

Trying to add up values and matching them to other tools leads to nothing but confusion.

Still, there should be some reliable process of checking server health with respect to memory usage.

Typical scenario - a fellow oracle administrator or a monitoring tool alerts you that a server is running low on memory, which is unexpected because "nobody touched it" and "nobody changed anything". So far the server has been running without any problem for months. Oracle administrators claim THEIR databases are set correctly, the sum of their max SGAs and max instance PGAs are below usable memory, therefore there must be something wrong with YOUR system.

Basicaly, two things may go wrong really quick:
- system starts swapping and preformance goes down
- swap area is filled with reservations and the system becomes unresponsive as no more memory can be allocated by new processes and you are eventually even unable to log in, unless "something dies".

In a situation like this I need to find processes which consume memory, swap or both more then reasonably.

What do you do? How do you find out what is going wrong before the worst happens and the system crashes?

[Dennis Handly]

>How do you find out what is going wrong before the worst happens and the system crashes?

Monitor the total line in swapinfo -tam.
Look at vmstat(1) for the "po" stats.

[chris huys_4]

hints.

* The kmeminfo output is not taken on the same system, the swapinfo output is taken from. (i.e. the kmeminfo output is taken from a system with 12GB of physical ram, the swapinfo output is taken from a system with 16GB of physical ram)

* swapinfo output gives "swap related info", which doesnt have a immediate correspondence with "kmeminfo without options" output that "gives" in the first place "memory related info"

*kmeminfo -user output "bridges" "swap and memory related info", as it gives detailed swap and memory related output for each of the processes executing on the system, put it in a cronjob and let it execute every hour or so..

* add more deviceswap, the reserved deviceswap, 8724, is to close to the total amount of device swap that is defined on the system (10GB), besides it would be better if you have only 2 device swapspaces, the current 4Gbyte for lvol2 and a second device swap of 17GB (16Gb + some more (i.e. 1Gb)), a second device swap, that can also double as primary dump space in case you need to save a "complete crashdump" and want to shorten the amount time needed to let the system boot up after a "complete crashdump" was "taken"

* add the vmstat 1 3600 in the same cronjob also like dennis said.. and a swapinfo -tm, and maybe also add a crashinfo -c /dev/kmem /stand/vmunix also, to be able to crosscheck some of the swapvalues of swapinfo -tm

* I would even add a >> top -f /tmp/top.out -n to the cronjob, the more info the better..

[Petr Wünsch]

Thanks for all your hints.