- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: Find memory leak
Categories
Company
Local Language
Forums
Discussions
- Integrity Servers
- Server Clustering
- HPE NonStop Compute
- HPE Apollo Systems
- High Performance Computing
Knowledge Base
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Knowledge Base
Forums
Discussions
- Cloud Mentoring and Education
- Software - General
- HPE OneView
- HPE Ezmeral Software platform
- HPE OpsRamp Software
Knowledge Base
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 06:21 AM
05-08-2002 06:21 AM
Find memory leak
My swapinfo -ta gives this out put
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 0 1048576 0% 0 - 1 /dev/vg00/lvol2
dev 8884224 0 8884224 0% 0 - 1 /dev/vg07/lvol2
reserve - 1314528 -1314528
memory 6501260 2696656 3804604 41%
total 16434060 4011184 12422876 24% - 0 -
Why is that alway reserved being used and both DEV is 0% used..
My swapinfo -ft shows 100 % used
Kb Kb Kb PCT Kb Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
total 0 0 0 100% - 0 -
Could some one explain me this please.
ENV: N-Class, HPUX 1100, Oracle 8i
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 06:31 AM
05-08-2002 06:31 AM
Re: Find memory leak
See
http://bizforums.itrc.hp.com/cm/QuestionAnswer/1,,0x08830559ff7cd4118fef0090279cd0f9,00.html
Steve steel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 06:35 AM
05-08-2002 06:35 AM
Re: Find memory leak
Device swap is not being used because you've got swapmem_on set to 1 and there's plenty of spare memory.
Your server has 8Gb RAM and you need about 4Gb of swap space.
swapinfo -ft displays information about filesystem swap, which you're not using. 0 of 0 could be argued as being 100% but I reckon that's a bugette in swapinfo.
Regards,
John
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 06:37 AM
05-08-2002 06:37 AM
Re: Find memory leak
FYI. This is useful for leaks
#!/usr/bin/ksh
#
# Show processes sorted by size of core image
#
# Usage:
# psram [ quantity ]
#
# where quantity is the top RAM processes to show (default is 20)
#
set -u
if [ $# -gt 0 ]
then
TOPPROCS=$1
else
TOPPROCS=20
fi
MYNAME=$(basename $0)
TEMPFILE=/var/tmp/$MYNAME.$$
trap `rm -f $TEMPFILE > /dev/null 2>&1` 0 1 2 3 15
UNIX95= ps -e -o ruser,vsz,pid,args > $TEMPFILE
head -1 $TEMPFILE
DASH5="-----"
DASH25="$DASH5$DASH5$DASH5$DASH5$DASH5"
echo "$DASH5---- $DASH5- $DASH5 $DASH25$DASH25"
grep -v "VSZ COMMAND" $TEMPFILE | cut -c -78 | sort -rn -k2 | head -${TOPPROCS}
rm $TEMPFILE > /dev/null 2>&1
#### END OF SCRIPT
Steve Steel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 06:43 AM
05-08-2002 06:43 AM
Re: Find memory leak
UNIX95= ps -e -o ruser,sz,vsz,pid,ppid,args | grep -v '^RUSER' | sort -nr -k2,2 | head -10
Perhaps you can set up a cron job that appends this output to a log file every 5 minutes or so.
Or, you can run:
nohup glance -adviser_only -syntax /opt/perf/examples/adviser/proc_mem_leak -j 300 > glance.out &
You can also modify a copy of the syntax file /opt/perf/examples/proc_mem_leak if you want to customize the output.
The percentage of swap reservations is relevant only if you suspect that it might reach 100%. Basically, every process needs to *reserve* swap whether or not it needs to use it. If there is no swap left to reserve, the process won't start. Unless you see this percentage getting closer to something like 90%, you need not be concerned. But it is important to monitor.
The output from swapinfo -ft is not relevant in your case because you have no file system configured as swap. The percentage of 100% is confusing because it is really 0 Kb used out of 0 Kb available.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 07:01 AM
05-08-2002 07:01 AM
Re: Find memory leak
Thanks for all quick responces.
OK, My worry is one my back up process (Oracle Hotbackup) worse over time until system is rebooted. I was thinking could be some memory leak or bottle neck.
Well, now swapinfo output.
Is that mean I don't have any issue with swap memory..?
John..
yes..swap_mem is ON. Is it OK..? or Do I need it changed.
I was glancing glance on the system table screen the nbuff is little confusing. I've dbc_max set to 8 and dbc_min is 5. nbuff is set to 0.
System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4000 606 15 15
File Table (nfile) 20010 18012 90 90
Shared Mem Table (shmmni) 128 10 8 8
Message Table (msgmni) 128 2 2 2
Semaphore Table (semmni) 128 27 21 21
File Locks (nflocks) 900 395 44 44
Pseudo Terminals (npty) 512 7 1 1
Buffer Headers (nbuf) na 222222 na na
Steve,
From your script it gives list of the process with big VSZ. How do you pinpoint what is wrong.
Mladen,
I am collecting advisory output. And getting some disk bottleneck but not the memory bottleneck
I'll assign points shortly.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 07:43 AM
05-08-2002 07:43 AM
Re: Find memory leak
The memory line in the output from 'swapinfo -ta' is about pseudo-swap. You turn pseudo-swap on by setting the kernel parameter swapmem_on=1.
You can turn pseudo-swap off by setting swapmem_on=0.
In your case, pseudo-swap is at 41% utilization, which is ok. However, if this value gets close to 100%, you should be concerned. Should that happen, and if you have plenty of disk space, just disable pseudo-swap and enable more device swap. Indeed, we have had problems with Sybase using up pseudo-swap. The recommendation from both HP and Sybase was to disable pseudo-swap. HP recommended the same for Oracle database.
The Glance adviser syntax /opt/perf/examples/proc_mem_leak is designed for detecting memory leaks. It won't detect memory bottlenecks.
If you want to monitor a specific process, you can create your own adviser syntax and run it as mentioned before (use PROCESS LOOP) See /opt/perf/paperdocs/gp/C/adviser.pdf for more examples and details.
It seems to me that you should increase nfile. I would double it, at least.
The nbuf is a bit confising because you have configured dynamic buffer cache. It shouldn't be a concern. The only thing I would change is dbc_max_pct and dbc_min_pct. With dbc_max_pct equal to 8, you buffer cache may grow up to 8% of your physical memory. So, if you have 8Gb, your buffer cache may grow up to 640 Mb. HP recommends that you don't need more than about 300 Mb of buffer cache. Ultimately, you want to monitor GBL_MEM_CACHE_HIT_PCT
in Glance. If you lower dbc_max_pct and this metric is at least 80-90% on average,
it is ok. The moment GBL_MEM_CACHE_HIT_PCT starts show significat decrease, you should then increase dbc_max_pct. But the way things look right now, it is too high.
Here is some info on Buffer Headers (from /opt/perf/paperdocs/gp/C/metrics.txt):
TBL_BUFFER_HEADER_AVAIL
--------------------
The configured number of headers pointing to buffers in the file
system buffer cache. This can be set by the nbuf kernel
configuration parameter. nbuf is used to determine the maximum
total number of buffers on the system.
These are used to manage the buffer cache, which is used for all
block IO operations. When nbuf is zero, this value depends on
the bufpages size of memory (see System Administration Tasks
manual). A value of "na" indicates either a dynamic buffer cache
configuration, or the nbuf kernel parameter has been left
unconfigured and allowed to float with the bufpages parameter.
This is not a maximum available value in a fixed buffer cache
configuration. Instead, it is the initial configured value. The
actual number of used buffer headers can grow beyond this initial
value.
TBL_BUFFER_HEADER_USED
--------------------
The number of buffer headers currently in use.
This dynamic value will rarely change once the system boots.
During the system bootup, the kernel allocates a large number of
buffer headers and the count is likely to stay at that value
after the bootup completes. If the value increases beyond the
initial boot value, it will not decrease. Buffer headers are
allocated in kernel memory, not user memory, and therefore, will
not decrease.
This value can exceed the available or configured number of
buffer headers in a fixed buffer cache configuration.
This metric is updated every 30 seconds or the sampling interval,
whichever is greater.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 07:50 AM
05-08-2002 07:50 AM
Re: Find memory leak
Based on your swapinfo output, it doesn't look like you have any memory pressures.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 08:51 AM
05-08-2002 08:51 AM
Re: Find memory leak
Mladen,
Here is the memory report.
Cumm page/out increases every time screen refreshes.
MEMORY REPORT Users= 10
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 1461 12043 324.6 272.4 704.8
Page In 555 4450 123.3 100.6 265.9
Page Out 1 13 0.2 0.2 0.7
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 4kb 52kb 0.8 1.1 3.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 2 25 0.4 0.5 1.5
Total VM : 999.6mb Sys Mem : 376.9mb User Mem: 3.04gb Phys Mem: 8.00gb
Active VM: 309.8mb Buf Cache: 655.4mb Free Mem: 3.95gb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 09:14 AM
05-08-2002 09:14 AM
Re: Find memory leak
You have plenty of free memory (almost 4 Gb).
There is no memory bottleneck.
I would set dbc_max_pct=4
and dbc_min_pct=1
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 10:28 AM
05-08-2002 10:28 AM
Re: Find memory leak
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 11:06 AM
05-08-2002 11:06 AM
Re: Find memory leak
Which symptoms are telling you that your hotbackup is getting worse over time? What exectly is the process name? Just 'hotbackup' ?
Regards ... Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 11:47 AM
05-08-2002 11:47 AM
Re: Find memory leak
We run a perl script to take backup of our oracle database
This script actually takes each table spaces and copy it over to a different filesystem and then compress it. This normally runs for five hours but gradualy this gets worse and backup runs for more than 12 hours. This is the only concern I've Otherwise system responce is quite good. I want to know what is casuing this. Reboot fixes this problem for some time but then slowly time taken for backup increases. It's really become a mistry to me solve this. I'm collecting all kind of information but not been able to make out who is the culprit.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 12:42 PM
05-08-2002 12:42 PM
Re: Find memory leak
The only obvious reason at this point would be the kernel parameter 'nfile'. After a system reboot, the number of open files will increase over time (especially if you have other runaway processes). So, at some point, it is possible that your "hotbackup" is hitting the maximum number of open files (as set by nfile) and so, it is unable to open more files. It is possible that it has to wait for another process to close files.
Your current nfile utilization is at 90%. I think you should definitely increase the value of this kernel parameter.
There are other kernel parameters that should be checked (Glance only shows a few). If there are any recommendations from Oracle, I would check those first.
If this doesn't help, try posting your perl script on the forum.
Mladen
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 12:48 PM
05-08-2002 12:48 PM
Re: Find memory leak
if a process is unable to open a file due to 100% nfile utilization. So, check the syslog!
But, in any case, do increase the value of nfile.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 01:02 PM
05-08-2002 01:02 PM
Re: Find memory leak
By default, nfile depends on maxusers via a formula:
NFILE=16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL)
In most cases, you just need to increase the value of maxusers. The value of nfile will be increased automatically.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-08-2002 01:45 PM
05-08-2002 01:45 PM
Re: Find memory leak
Yes. looks like nfile needs a raise in value!.
I'll do it next maintainance window.
Thanks for all the help.
Attached is the script.