How to find root cause of memory leak hpux 11.11

[Frank de Vries]

We have a memory leak .
We have hp900 L4000 model with hpux 11.11 (pa-risc). We have 4Gbyte of memory.
There are 3 oracle instance running 10.2.03 of SGA size 160MB each. After startup we have about 1,7Gb free (top). And swapinfo -a , shows very low activity.

After a couple of days, the memory free is down to under 100MB and swapinfo -a shows PCT 60% or more.
on
We did have emctl console running, but I have disabled that to safe memory, but it is just a matter of time before memory is depleted.

The Oracle instances have been there for 18months and only recently we have this. There were no changes on the db. Maybe we should have installed a later release 10.2.04 ?
Still, why did n't we have it then from outset ?

I am in de dark. Any tips how I can troubleshoot in more depth where this is comming from ?

[Laurent Menase]

is it kernel memory which raises or user memory?
you can check with glance as a first approach
-> if it is user memory , then you need to look what application is using more memory
-> if it is kernel memory,
Does it goes to the point where the system can't work anymore?

If you have a kernel memory leak, the best is to contact HP support.

[Turgay Cavdar]

Type:
# glance -m
and see the values of Sys Mem/User Mem/Buf Cache. You should also consider dbc_max_pct value. If you have kmeminfo tool you can go further analysis.

[Hakki Aydin Ucar]

if need an description of memory leak that is the amount of memory that they occupy keeps growing and growing; have to find you experienced application type OR OS type memory leaks ? apparently you faced application type but if you can check this out for OS memory leak;

+ Check for known OS Memory Leaks with an unsupported utility called kmeminfo The Response Center can assist with using the output of kmeminfo. (o The JFS inode cache, is sized by the vx_ninode kernel parameter. vx_ninode is determined by size of RAM that an 11.11 system with 8GB of memory, vx_ninode is defaulted to 256,000 )

additionaly check this out:

+ If you see a process called mib2agt using an excessive amount of memory, this binary, mib2agt, has a known memory leak fixed in an 11.X patch called PHSS_27858 .

[Frank de Vries]

Although it is well meant, your answers are sofar not all that usefull.

First I don't appear to have glance (installed ?)

And kmeminfo returns figures that dont add up.
It says our physical memory is 1GB , while
xctm and dmesg say it is 4GB (and that is also what is in the machine)

SO what does this kmeminfo actual learns us:
(see output)

[orasrv4:/]# /usr/contrib/Q4/bin/kmeminfo
see attached

[Duncan Edmonstone · Accepted solution]

Frank,

The numbers reported by kmeminfo are in pages of 4K, so that does add up to 4GB.

As you don't have glance, you probably also don't have Performance Agent (used to be called Measureware) - you can check by trying the following command:

mwa status

If that returns a "not found" message, you don't have Performance Agent installed.

Assuming you don't, you have no historical data on what processes are consuming memory and how much, so you'll have to start from scratch.

What you probably need to do is look at the top X processes consuming memory over time, so you'll need some sort of cron job that gathers info like this:

ps -elf | awk '{printf "%6s %10s %8s %s\n",$4,$3,$10,$15}' | sort -rnk3 | head -20

which looks at the top 20 memory consumers over time.

From that you _might_ see which (if any) processes are consuming more memory.

Bare in mind though that the way memory management works, you'll never see the memory consumed by a process going down until there is actually memory pressure on the system... by which I mean that when processes allocate memory by calls to system calls like malloc, the process size will grow, but when a call to free is made for that memory you won't see the process size of the process shrink again unless there is actual memory pressure on the system...

HTH

Duncan

[Turgay Cavdar]

Hi,
kmeminfo shows units in pages so you have to multiply the values by page size (4KB).
Your buffer cache is nearly 1,3GB. Is it high? For oracle installs yes, you should consider to decrease dbc_max_pct between 5 and 10...

[Dennis Handly]

>swapinfo -a, shows very low activity.

(You should use "swapinfo -tam" and look at the total line.)

If you use top, can you see any memory growth in a process?
This would indicate either a leak or heap fragmentation.

[john korterman]

Hi Frank,

you could take a look at the memory consumption of individual processes, e.g.:

$ UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

which shows a per user sorted list.

Or a top ten of memory consumers:

$ UNIX95= ps -e -o ruser,sz,vsz,pid,ppid,args | grep -v '^RUSER' | sort -nr -k2,2 | head -10

If you execute this at regular intervals, you may see some development in the consumption.
But does your system actually stop because of insufficient memory?


regards,
John K.

[Bill Hassell]

You might have a process with a memory leak but you might also have a number of new programs running. A memory leak is a programming, probably something you cannot fix without contacting the manufacturer of the software. Also, you can only see a memory leak after you establish what is normal (after a reboot). The above ps commands will show you the biggest processes which is useful, especially if you see something you don't recognize that is using a lot of RAM. If all looks OK, then run the ps command into a log file so you can compare previous results.