Process and Memory Management

[Sagar Sirdesai]

Hi Guys
I have rp4440 with 11.11 installed
It had 64 GB of RAm.

Actually I want learn Process management and Memory Management.
This server has a lot of JAVA Programms running . And the CPU and Memory Utilization is always nearing 100 % .
Right I check using vmstat command

There is lot of Page in's but no page Outs and also There are lot of page faults .
Could please help what does this scenario means

And how do I find out which process cosuming more CPU / Memroy .
Also what is page faults


Please help me .

[Antonio Cardoso_1 · Accepted solution]

Hi Sagar,

1) to get sorted list of CPU consumption per process, use top command.

2) to monitor CPU on a singl process, consider sar command.

3) to check memory usage, consider using:
export UNIX95=1
ps -ef -o pid,sz,vsz,comm,args

does this answer your questions?

[Antonio Cardoso_1]

also, regarding pages in/out, if your swapinfo reports no usage of device, it means that all processes are working in memory only, and not swaping out to device. This is great
exemple:
# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 3304292 -3304292
memory 14465736 1696364 12769372 12%

[Sagar Sirdesai]

Hi Antonio
Thanks for your help
Firstly
What is the differnce between between vsz and sz.
does sz -- Means total size of the process in memeory
vsz --- Means total size of the process.

Also
What command optins should i use fro sar to single out a process

I normally use

ps -e -o pid,pcpu | grep "desired pid"

When i use use swapinfo -tam command
I do see that my device files are used up

So is the reason that the Pageouts are not possible
because there is no space to pageout .

Also when there is a lot of pagein's does it heavily loads the CPU,



Thanks again

Sagar Sirdesai

[AwadheshPandey]

See attached document this will help you.

Awadhesh

[RAC_1]

Are Sagarwa,

1. First thing to is ask users whey so many Java processes are to be run. With some start options in java (particularly initial heap size, max heap size etc.), you can control how much memory they are using and should be using.
ps -xef | grep [j]ava will tell you what options they are starting Java with.

2. Now, to know what programs are using most of the memory/cpu do as follows.
UNIX95= ps -ef -o vsz,pid,ppid,ruser,args | sort -k1
UNIX95= ps -ef -opcpu,pid,ppid,ruser,args|sort -k1

3. Page faults keep happening. Unless their number is very high, you should not be worrying about. It happens, when processes tries to access a memory page and it;s not there. It may be paged out to swap. This is page fault. Also check swap usage. swapinfo -mat. If you have this much RAM, what are your settings for following
dbc_max_pct, dbc_min_pct, nbufs, buffpages. Also what about swapmem_on kernel setting?


In nutshell, unless you ask users to control memory usage, you would always be hitting 100% CPU, memory.

[Antonio Cardoso_1]

Sagar,
additionally to previous answers, if your server is intensively running java applications, you should run hpjconfig utility on it (if not yet done) in order to check that patches and system configuration is OK for java:
http://www.hp.com/products1/unix/java/java2/hpjconfig/index.html

[Sagar Sirdesai]

Hi All
Thanks all to your answers....

Right now servers is stable . We figure out a few processes which make server task.
So we are not running them now.

Awadesh : size command looks very useful .
Please help me how do I use more effecyively.

RAC: here are the required kernel parameters.
bufpages 204800
nbuf 0 dbc_max_pct 0
dbc_min_pct 0
swapmem_on 1

Please assist and advise.
Regards
Sagar

[RAC_1]

Kernel parameters look fine. You have 838MB of buffer space, which good enough. As said earlier, the real trick is control users, particularly java like users. Also make sure that you are up to date on Java pacthes for good java performance.

[Sagar Sirdesai]

Hi RAC

thanks for the tips

One last help
Could you please help me understanding the kernel parameters .So that I can use them more appropriately .

I am aware of swapmem_on .

[RAC_1]

Which kernel parameters you want help with?

[Sagar Sirdesai]

bufpages
nbuf
dbc_max_pct
dbc_min_pct
maxdsiz
maxdsiz_64
maxtsiz
maxtsiz_64
maxssiz
maxssiz

Also what are data size , test size , stack size of process .
It will be more clear if any exaple is used to explain.

Thanks again
Sagar

[RAC_1]

Refer following link for detailed explanation.

http://docs.hp.com/en/TKP-90202/index.html
http://docs.hp.com/en/939/KCParms/KCparams.OverviewAll.html

bufpages - buffer pages setting. Buffer cache setting.
nbuf - no. of buffers. One of bufpages or nbuf needs to set. This static buffer cache setting.
dbc_max_pct - Used for dynamic buffer cahe max setting. Need to give max % of RAM that will be used as buffer cache.
dbc_min_pct - min % dynamic buffer cache. Dynamic buffer cache can expand, contract as required. With static, you set once and that memory set aside as buffer cache.
maxdsiz - data size for 32 bit programs.
maxdsiz_64 - data size for 64 bit programs.
maxtsiz - text segment size for 32 bit programs
maxtsiz_64 - text segment size for 64 bit programs
maxssiz - stack segment size for 32 bit programs.
maxssiz_64bit - stack segment size for 32 bit programs.

Every program has three segments assigned to it-data,text and stack.

[Bill Hassell]

In answer to your earlier questions:

> does sz -- Means total size of the process in memeory
> vsz --- Means total size of the process.

The man page for ps is helpful here. sz is the core image of the process measured in pages while vsz is the same measurement shown in Kbytes.

> What command optins should i use fro sar to single out a process

sar has no options for a single process (per the man page). sar measures overall kernel activity.

> I normally use
> ps -e -o pid,pcpu | grep "desired pid"

Again, the man page is very useful. You can show this information without grep:

UNIX95= ps -e -p "desired pid" -o pid,pcpu

I noticed in your example that UNIX95 was not set on the command line. If you use oprions like -o -H -C you will get an error unless UNIX95 is set. However, it is a very bad idea to set or export UNIX95 in your shell as it affects many other commands and libraries. Be sure UNIX95 is unset in your shell and always set it temporarily on the command line. You can safely set UNIX95 for all ps commands with an alias in .profile:

alias ps="UNIX95= ps"

> When i use use swapinfo -tam command I do see that my device files are used up

If you mean that the percentage of occupied space is close to 100%, then your RAM is far too small for the applications.

> So is the reason that the Pageouts are not possible because there is no space to pageout .

Pageouts only occur when processes are deactivated to make room for other processes. So this is an instantaneous measurement and does not show previous pageouts. Note that extensive use of memory mapped files can also occupy swap space. If there is no space to pageout, the processes needing more RAM will fail or crash.

> Also when there is a lot of pagein's does it heavily loads the CPU,

Pagein is actually two conditions measured as one. When a process starts for the first time, the executable file is the source and the process is 'paged in' from that file. So every program start will create pageins. If a process must be returned from swap, that will also cause pageins. There is no way to separate the count into starts versus pagein from swap.

Now pageins do not load the CPU as there is little computation beign done. But as is true for swapping of any sort, the entire system will be crippled (as far as performance) when a lot of pageouts start occuring.

You will find Glance exceedingly helpful in looking at specific processes as well as instantly determining the busiest processes (RAM, CPU, disk I/O).

[Sagar Sirdesai]

Hi Gurus
I apologize for the late response.
Thanks every one for your response.

One more query I have regarding required memory for a program
With respect to a process Is it possible to find what values I need set for parameters maxdsiz, maxtsiz , maxssiz so that the process does not exit because of lack of memory.

I checked the size dommand .. But was not successful ..
Please suggest

Sagar

[Bill Hassell]

The kernel parameters maxdsiz, etc are simply protection limits for bad programs. Setting the limits to very high values will eliminate ONE of the limits for memory allocation. Simply put, you can increase maxdsiz to about 1800 megs, and maxdsiz_64 to 8Gb. (Note that maxdsiz_64 must always be larger than maxdsiz). maxssiz and maxtsiz never need changing except for most systems.

Memory management is not simple, especially if you have to run old 32bit applications. With 64Gb of RAM, you should not have to be concerned about running out of RAM, but there are several limits imposed by addressing in 32bit programs. These limits will appear to be out of memory but the reason is due to the program itself, not the OS and available RAM.

[Sagar Sirdesai]

Hi Bill

How do I know whether a program is a bad program .. To start with I need to specify some benchmark for java process .

In my case I set values for Kernel Parameters

Asked the users to start thier builds but the builds fail with Out of Swap error.
I checked I do have swap space ..
Then I increase the value of mazdsiz and so-on till the build gets successful.
Can I find out any how the kernel values required for a java process and then set the kernel parameters accordingly

Thanks again
Sagar

[Bill Hassell]

> How do I know whether a program is a bad program ..

A bad program is one that uses massive resources such as memory and/or disk I/O without a good reason, or the resources are consumed due to mistakes in the code. Poor programming would also include a lack of requirements, that is, the programmer does not know how much memory or disk I/O is needed.

> To start with I need to specify some benchmark for java process.

Java is a complex language to analyze, partly because it can be used to generate dozens to hundreds of threads or subprograms (and hitting limits with kernel parameters such as NPROC, NFILE and threads). A lot depends on the quality of the Java support libraries which means that the Java installation must be patched to the latest level, Java 5.something at this point. See:

http://www.hp.com/products1/unix/java/?jumpid=go/java

A programmer can easily request 1000 megs of RAM when only 100 megs is needed. The above recommendations to identify such processes (using the ps command) can point out big resource problems.

> In my case I set values for Kernel Parameters

As far as limits such as maxdsiz, nproc, nfile, maxuproc, etc, they can be set to very high values to allow the programs to run, then the programmers can work on improving the code for more efficient use of system resources.

> Asked the users to start thier builds but the builds fail with Out of Swap error.
I checked I do have swap space ..

Well, all HP-UX systems have swap space, the question is how much. HP-UX is a virtual memory system so memory is a combination of RAM and swap. Once you run out of RAM, HP-UX will use the swap space. If you have 4Gb of RAM but all your programs need a total of 16Gb, then you'll need more than 12Gb of swap space. However, swap is extremely slow compared to RAM so your programs will run very slow when swapping starts. There is no fix for this except a lot more RAM. To avoid swapping, RAM should be large enough to keep swapping below 4-8 pages per second.

As far as the builds that are failing, the simplest fix is to double or triple your swap area. Use the command swapinfo -tam to identify the current usage.

[Sagar Sirdesai]

Hi All
Thanks all for your comments

Sagar