topic Re: Find memory leak in Operating System - HP-UX

Find memory leak

Rushank — Wed, 08 May 2002 13:21:57 GMT

I want to findout a possible memory leak. I've glance but not sure how to pinpoint reason for memory leak.

My swapinfo -ta gives this out put
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 0 1048576 0% 0 - 1 /dev/vg00/lvol2
dev 8884224 0 8884224 0% 0 - 1 /dev/vg07/lvol2
reserve - 1314528 -1314528
memory 6501260 2696656 3804604 41%
total 16434060 4011184 12422876 24% - 0 -

Why is that alway reserved being used and both DEV is 0% used..

My swapinfo -ft shows 100 % used
Kb Kb Kb PCT Kb Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
total 0 0 0 100% - 0 -

Could some one explain me this please.
ENV: N-Class, HPUX 1100, Oracle 8i

Re: Find memory leak

Steve Steel — Wed, 08 May 2002 13:31:54 GMT

Hi

See

http://bizforums.itrc.hp.com/cm/QuestionAnswer/1,,0x08830559ff7cd4118fef0090279cd0f9,00.html

Steve steel

Re: Find memory leak

John Palmer — Wed, 08 May 2002 13:35:04 GMT

Hi,

Device swap is not being used because you've got swapmem_on set to 1 and there's plenty of spare memory.

Your server has 8Gb RAM and you need about 4Gb of swap space.

swapinfo -ft displays information about filesystem swap, which you're not using. 0 of 0 could be argued as being 100% but I reckon that's a bugette in swapinfo.

Regards,
John

Re: Find memory leak

Steve Steel — Wed, 08 May 2002 13:37:24 GMT

Hi

FYI. This is useful for leaks

#!/usr/bin/ksh
#
# Show processes sorted by size of core image
#
# Usage:
# psram [ quantity ]
#
# where quantity is the top RAM processes to show (default is 20)
#
set -u
if [ $# -gt 0 ]
then
TOPPROCS=$1
else
TOPPROCS=20
fi

MYNAME=$(basename $0)
TEMPFILE=/var/tmp/$MYNAME.$$
trap `rm -f $TEMPFILE > /dev/null 2>&1` 0 1 2 3 15

UNIX95= ps -e -o ruser,vsz,pid,args > $TEMPFILE
head -1 $TEMPFILE
DASH5="-----"
DASH25="$DASH5$DASH5$DASH5$DASH5$DASH5"
echo "$DASH5---- $DASH5- $DASH5 $DASH25$DASH25"
grep -v "VSZ COMMAND" $TEMPFILE | cut -c -78 | sort -rn -k2 | head -${TOPPROCS}
rm $TEMPFILE > /dev/null 2>&1
#### END OF SCRIPT

Steve Steel

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 13:43:55 GMT

To monitor for memory leaks, you can log the output from:

UNIX95= ps -e -o ruser,sz,vsz,pid,ppid,args | grep -v '^RUSER' | sort -nr -k2,2 | head -10

Perhaps you can set up a cron job that appends this output to a log file every 5 minutes or so.

Or, you can run:

nohup glance -adviser_only -syntax /opt/perf/examples/adviser/proc_mem_leak -j 300 > glance.out &

You can also modify a copy of the syntax file /opt/perf/examples/proc_mem_leak if you want to customize the output.

The percentage of swap reservations is relevant only if you suspect that it might reach 100%. Basically, every process needs to *reserve* swap whether or not it needs to use it. If there is no swap left to reserve, the process won't start. Unless you see this percentage getting closer to something like 90%, you need not be concerned. But it is important to monitor.

The output from swapinfo -ft is not relevant in your case because you have no file system configured as swap. The percentage of 100% is confusing because it is really 0 Kb used out of 0 Kb available.

Re: Find memory leak

Rushank — Wed, 08 May 2002 14:01:07 GMT

Thanks for all quick responces.

OK, My worry is one my back up process (Oracle Hotbackup) worse over time until system is rebooted. I was thinking could be some memory leak or bottle neck.

Well, now swapinfo output.

Is that mean I don't have any issue with swap memory..?
John..
yes..swap_mem is ON. Is it OK..? or Do I need it changed.
I was glancing glance on the system table screen the nbuff is little confusing. I've dbc_max set to 8 and dbc_min is 5. nbuff is set to 0.
System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4000 606 15 15
File Table (nfile) 20010 18012 90 90
Shared Mem Table (shmmni) 128 10 8 8
Message Table (msgmni) 128 2 2 2
Semaphore Table (semmni) 128 27 21 21
File Locks (nflocks) 900 395 44 44
Pseudo Terminals (npty) 512 7 1 1
Buffer Headers (nbuf) na 222222 na na

Steve,
From your script it gives list of the process with big VSZ. How do you pinpoint what is wrong.

Mladen,

I am collecting advisory output. And getting some disk bottleneck but not the memory bottleneck

I'll assign points shortly.

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 14:43:41 GMT

Rushank,

The memory line in the output from 'swapinfo -ta' is about pseudo-swap. You turn pseudo-swap on by setting the kernel parameter swapmem_on=1.
You can turn pseudo-swap off by setting swapmem_on=0.

In your case, pseudo-swap is at 41% utilization, which is ok. However, if this value gets close to 100%, you should be concerned. Should that happen, and if you have plenty of disk space, just disable pseudo-swap and enable more device swap. Indeed, we have had problems with Sybase using up pseudo-swap. The recommendation from both HP and Sybase was to disable pseudo-swap. HP recommended the same for Oracle database.

The Glance adviser syntax /opt/perf/examples/proc_mem_leak is designed for detecting memory leaks. It won't detect memory bottlenecks.
If you want to monitor a specific process, you can create your own adviser syntax and run it as mentioned before (use PROCESS LOOP) See /opt/perf/paperdocs/gp/C/adviser.pdf for more examples and details.

It seems to me that you should increase nfile. I would double it, at least.

The nbuf is a bit confising because you have configured dynamic buffer cache. It shouldn't be a concern. The only thing I would change is dbc_max_pct and dbc_min_pct. With dbc_max_pct equal to 8, you buffer cache may grow up to 8% of your physical memory. So, if you have 8Gb, your buffer cache may grow up to 640 Mb. HP recommends that you don't need more than about 300 Mb of buffer cache. Ultimately, you want to monitor GBL_MEM_CACHE_HIT_PCT
in Glance. If you lower dbc_max_pct and this metric is at least 80-90% on average,
it is ok. The moment GBL_MEM_CACHE_HIT_PCT starts show significat decrease, you should then increase dbc_max_pct. But the way things look right now, it is too high.

Here is some info on Buffer Headers (from /opt/perf/paperdocs/gp/C/metrics.txt):

TBL_BUFFER_HEADER_AVAIL
--------------------
The configured number of headers pointing to buffers in the file
system buffer cache. This can be set by the nbuf kernel
configuration parameter. nbuf is used to determine the maximum
total number of buffers on the system.

These are used to manage the buffer cache, which is used for all
block IO operations. When nbuf is zero, this value depends on
the bufpages size of memory (see System Administration Tasks
manual). A value of "na" indicates either a dynamic buffer cache
configuration, or the nbuf kernel parameter has been left
unconfigured and allowed to float with the bufpages parameter.

This is not a maximum available value in a fixed buffer cache
configuration. Instead, it is the initial configured value. The
actual number of used buffer headers can grow beyond this initial
value.

TBL_BUFFER_HEADER_USED
--------------------
The number of buffer headers currently in use.

This dynamic value will rarely change once the system boots.
During the system bootup, the kernel allocates a large number of
buffer headers and the count is likely to stay at that value
after the bootup completes. If the value increases beyond the
initial boot value, it will not decrease. Buffer headers are
allocated in kernel memory, not user memory, and therefore, will
not decrease.

This value can exceed the available or configured number of
buffer headers in a fixed buffer cache configuration.

This metric is updated every 30 seconds or the sampling interval,
whichever is greater.

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 14:50:51 GMT

If you are concerned about memory bottleneck, the main thing is to check the memory report in Glance. Look for page outs and deactivations. Some page outs are normal, but the cumulative amount should be very small. Any deactivations would indicate memory pressure.

Based on your swapinfo output, it doesn't look like you have any memory pressures.

Re: Find memory leak

Rushank — Wed, 08 May 2002 15:51:14 GMT

Mladen,

Here is the memory report.
Cumm page/out increases every time screen refreshes.

MEMORY REPORT Users= 10
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 1461 12043 324.6 272.4 704.8
Page In 555 4450 123.3 100.6 265.9
Page Out 1 13 0.2 0.2 0.7
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 4kb 52kb 0.8 1.1 3.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 2 25 0.4 0.5 1.5

Total VM : 999.6mb Sys Mem : 376.9mb User Mem: 3.04gb Phys Mem: 8.00gb
Active VM: 309.8mb Buf Cache: 655.4mb Free Mem: 3.95gb

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 16:14:01 GMT

This is normal.
You have plenty of free memory (almost 4 Gb).
There is no memory bottleneck.

I would set dbc_max_pct=4
and dbc_min_pct=1

Re: Find memory leak

Rushank — Wed, 08 May 2002 17:28:44 GMT

I'll make kernel changes during our maintainance window. But my problem of hotbackup getting worse over time and why a reboot fixes it.

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 18:06:37 GMT

Rushank,

Which symptoms are telling you that your hotbackup is getting worse over time? What exectly is the process name? Just 'hotbackup' ?

Regards ... Mladen

Re: Find memory leak

Rushank — Wed, 08 May 2002 18:47:01 GMT

We run a perl script to take backup of our oracle database
This script actually takes each table spaces and copy it over to a different filesystem and then compress it. This normally runs for five hours but gradualy this gets worse and backup runs for more than 12 hours. This is the only concern I've Otherwise system responce is quite good. I want to know what is casuing this. Reboot fixes this problem for some time but then slowly time taken for backup increases. It's really become a mistry to me solve this. I'm collecting all kind of information but not been able to make out who is the culprit.

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 19:42:06 GMT

Rushank,

The only obvious reason at this point would be the kernel parameter 'nfile'. After a system reboot, the number of open files will increase over time (especially if you have other runaway processes). So, at some point, it is possible that your "hotbackup" is hitting the maximum number of open files (as set by nfile) and so, it is unable to open more files. It is possible that it has to wait for another process to close files.

Your current nfile utilization is at 90%. I think you should definitely increase the value of this kernel parameter.

There are other kernel parameters that should be checked (Glance only shows a few). If there are any recommendations from Oracle, I would check those first.

If this doesn't help, try posting your perl script on the forum.

Mladen

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 19:48:51 GMT

If I recall correctly, there should be a message in /var/adm/syslog/syslog.log
if a process is unable to open a file due to 100% nfile utilization. So, check the syslog!

But, in any case, do increase the value of nfile.

Re: Find memory leak

Mladen Despic — Wed, 08 May 2002 20:02:10 GMT

NOTE:

By default, nfile depends on maxusers via a formula:

NFILE=16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL)

In most cases, you just need to increase the value of maxusers. The value of nfile will be increased automatically.

Re: Find memory leak

Rushank — Wed, 08 May 2002 20:45:19 GMT

Yes. looks like nfile needs a raise in value!.
I'll do it next maintainance window.
Thanks for all the help.
Attached is the script.