Memory Hog?

[Justin Willoughby]

I am trying to figure out if I have a memory leak in one of the processes running on a HP/UX 10.20 box or not.

Glance displays the following:

--------------------------------------------------------------------------------
PROCESS LIST Users= 1990
B3692A GlancePlus B.10.31 11:32:10 hostname 9000/899 Current Avg High
--------------------------------------------------------------------------------
CPU Util S SU U | 69% 56% 69%
Disk Util F F | 92% 96% 100%
Mem Util S SU UB B | 94% 94% 94%
Swap Util U UR R | 49% 49% 49%
--------------------------------------------------------------------------------

I know my disk I/O is quite high but there is not much I can do about that right now.

This system is a K570 with three CPUs and 1.5GB of memory.

sar ?m 1 10 reports the following:

HP-UX hostname B.10.20 U 9000/899 08/30/01

11:38:18 msg/s sema/s
11:38:19 0.00 32.67
11:38:20 0.00 179.00
11:38:21 0.00 27.00
11:38:22 0.00 207.00
11:38:23 0.00 53.54
11:38:24 0.00 195.05
11:38:25 0.00 63.00
11:38:26 0.00 146.46
11:38:27 0.00 84.16
11:38:28 0.00 152.00

Average 0.00 113.99


What should I check? How can I tell if one or two processes are taking up too much memory?

Thanks,

- Justin

[James R. Ferguson]

Hi Justin:

A god way to monitor memory growth at the process level (for potential leaks) is this:

# UNIX95= ps -e -o "user,vsz,pid,ppid,args"|sort -rnk2|more

Note the use of the POSIX (UNIX95) option of the 'ps' command to assist in this tracking, ranking the output in descending kilobyte core size. Note carefully that a blank character follows the 'UNIX95' variable declaration and that the 'ps' command begins without any interceding delimiter. Thus, the variable UNIX95 is set only for the one command line.

Regards!

...JRF...

[Justin Willoughby]

James,

Your command line suggestion displays the process using the most memory first (top)? Can I tell how much memory each processes is using?

The first few processes listed with your command are:

root 9796 1332 1 /opt/pd/lbin/pdclientd
root 7504 985 1 /opt/dce/sbin/rpcd
root 6848 20422 20421 /opt/perf/bin/alarmgen -svr 20421 -t alarmgen /var/opt/perf/
root 6440 1029 1 basicdsd
root 5500 20345 20340 /opt/perf/bin/rep_server -t SCOPE /var/opt/perf/datafiles/lo
root 5168 28571 1 /usr/adsm/dsmc sched
root 4404 20421 20340 /opt/perf/bin/agdbserver -t alarmgen /var/opt/perf/datafiles

Any thing look unusual here?

- Justin

[linuxfan]

Hi Justin,

you could try something like

UNIX95= ps -e -o vsz=Kbytes -o ruser,pid,ppid,args|sort -rnk2 |more

or you could put this as an alias in your .kshrc

psmem="UNIX95= ps -e -o vsz=Kbytes -o ruser,pid,ppid,args|sort -rnk2 |more"

-HTH
Ramesh

[Sridhar Bhaskarla]

As you have MWA installed, you can configure workloads on your system and collect the information. I found it very useful for my oracle database. Check out the syntax for
the configuration in /var/opt/perf/parm file. Once you start collecting the data specific to the processes, you can easily find if there is a problem by measuring the following metrics

* APP_PRI_WAIT_PCT
* APP_DISK_SUBSYSTEM_WAIT_PCT
* APP_MEM_WAIT_PCT
* APP_SEM_WAIT_PCT
* APP_NETWORK_SUBSYSTEM_WAIT_PCT
* APP_SLEEP_WAIT_PCT
* APP_IPC_SUBSYSTEM_WAIT_PCT

You have more metrics defined in reptall in the same directory.

I promise that you will definitely have fun with this analysis.

-Sri

[Justin Willoughby]

Sridhar,

How do I configure workloads?

Do I use extract to get the information I needed? I have not used WMA much. Is there a good tutorial for WMA that you know of?

Thanks,

- Justin

[Sridhar Bhaskarla · Accepted solution]

Justin,

It's not that difficult. If you have a look at /var/opt/perf/parm file, you can see some examples. Yes. You need to use extract to pull out the information.

For ex., I will configure my parm file like this

application = oracle
user = oracle

So, this will collect the metrics of all the processes running as oracle. Then I will prepare a report file with the following


DATE
TIME

APP_NAME
APP_PRI_WAIT_PCT #Gets the CPU WAIT
APP_MEM_WAIT_PCT #MEM WAIT PCT
AP_SEM_WAIT_PCT #PCT WAITING TO GET SEMAPHORE
APP_IPC_SUBSYSTEM_WAIT_PCT #PCT WAITED in IPC QUEUE

#END

Now I can run extract either interactively or non-interactively to get the information.

You try running extract manually with GUIDE option. Specify this file as the report file. It's easy.

The documents are there under /opt/perf/paperdocs.

-Sri
-Sri

[Justin Willoughby]

Ok, I used the metrics susgested by Sridhar to get the data from yesterday using extract, e.g.:

APP_PRI_WAIT_PCT
APP_DISK_SUBSYSTEM_WAIT_PCT
APP_MEM_WAIT_PCT
APP_SEM_WAIT_PCT
APP_NETWORK_SUBSYSTEM_WAIT_PCT
APP_SLEEP_WAIT_PCT
APP_IPC_SUBSYSTEM_WAIT_PCT

I have created some graphs of this data. I am not sure what I should be looking for or was is acceptable. If anyone can take a minute and take a look at this data/graphs and give me pointers on what I should be looking for and what is ok and not ok.

Thanks for all the replies so far! I am not sure why the Sleep Wait graph turned out the way it did.

- Justin

[Sridhar Bhaskarla]

These metrics give the %Time the processes are waiting to grab the resources. So, you will get a good feeling of how the systems are doing.

Coming back to your original question of finding out the processes that consume most of the memory, you need to pull some stats again for other metrics like APP_MEM_PRIVATE_UTIL, APP_MEM_VIRT etc to see if they are the ones that are causing the problems over the time.

You will get more granularity if you further devide your application definitions into small
segments. For ex., I will devide my oracle application like this

application=orawriter
file=ora_dbw*

application=oraarchiver
file=ora_arc*

Now I can analyze the system w.r.t each of these processes.

Please do not assign anymore points. There shouldn't be more than 10 points in a thread.

-Sri

[Justin Willoughby]

Sridhar, is that 10 points per person per thread?

I used the two metrics you noted, APP_MEM_PRIVATE_UTIL & APP_MEM_VIRT.

APP_MEM_PRIVATE_UTIL gives no data.
APP_MEM_VIRT gives four sets of data per time e.g.

00:00 774144.0
00:00 69632.0
00:00 24496.0
00:00 1868.0
00:05 773120.0
00:05 74752.0
00:05 27909.0
00:05 1868.0

What do the four numbers represent?

I take it you set the application definitions in the /var/opt/perf/parm file? This file is used for associating one or more executables with a name? I am guessing only new data collected with have this logged?

Thanks again!

- Justin

[James R. Ferguson]

Hi Justin:

No points for this -- just an opinion, here.

The guidelines for point assignment for multiple responses to a thread aren't terribly clear. We see everyting from 10-points awarded for a request to the author for more information, to no points awarded for multiple contributions by an individual as he/she struggles to provide problem resolution.

You are free to use your own judgement. The official guidelines exist here:

http://us-support.external.hp.com/estaff/bin/doc.pl/forward/screen=estaffAssistance/sid=a86fa1ac1b8c606413?Page=file0002#forpoints

I think that Sridhar's offer to limit the number of points collected in one post shows a professional who is more interested in solving a problem than in earning recognition -- a very admirable attribute.

Regards!

...JRF...