How to find root cause of memory leak hpux 11.11

[Frank de Vries]

We have a memory leak .
We have hp900 L4000 model with hpux 11.11 (pa-risc). We have 4Gbyte of memory.
There are 3 oracle instance running 10.2.03 of SGA size 160MB each. After startup we have about 1,7Gb free (top). And swapinfo -a , shows very low activity.

After a couple of days, the memory free is down to under 100MB and swapinfo -a shows PCT 60% or more.
on
We did have emctl console running, but I have disabled that to safe memory, but it is just a matter of time before memory is depleted.

The Oracle instances have been there for 18months and only recently we have this. There were no changes on the db. Maybe we should have installed a later release 10.2.04 ?
Still, why did n't we have it then from outset ?

I am in de dark. Any tips how I can troubleshoot in more depth where this is comming from ?

[Laurent Menase]

is it kernel memory which raises or user memory?
you can check with glance as a first approach
-> if it is user memory , then you need to look what application is using more memory
-> if it is kernel memory,
Does it goes to the point where the system can't work anymore?

If you have a kernel memory leak, the best is to contact HP support.

[Turgay Cavdar]

Type:
# glance -m
and see the values of Sys Mem/User Mem/Buf Cache. You should also consider dbc_max_pct value. If you have kmeminfo tool you can go further analysis.

[Hakki Aydin Ucar]

if need an description of memory leak that is the amount of memory that they occupy keeps growing and growing; have to find you experienced application type OR OS type memory leaks ? apparently you faced application type but if you can check this out for OS memory leak;

+ Check for known OS Memory Leaks with an unsupported utility called kmeminfo The Response Center can assist with using the output of kmeminfo. (o The JFS inode cache, is sized by the vx_ninode kernel parameter. vx_ninode is determined by size of RAM that an 11.11 system with 8GB of memory, vx_ninode is defaulted to 256,000 )

additionaly check this out:

+ If you see a process called mib2agt using an excessive amount of memory, this binary, mib2agt, has a known memory leak fixed in an 11.X patch called PHSS_27858 .

[Frank de Vries]

Although it is well meant, your answers are sofar not all that usefull.

First I don't appear to have glance (installed ?)

And kmeminfo returns figures that dont add up.
It says our physical memory is 1GB , while
xctm and dmesg say it is 4GB (and that is also what is in the machine)

SO what does this kmeminfo actual learns us:
(see output)

[orasrv4:/]# /usr/contrib/Q4/bin/kmeminfo
see attached

[Duncan Edmonstone · Accepted solution]

Frank,

The numbers reported by kmeminfo are in pages of 4K, so that does add up to 4GB.

As you don't have glance, you probably also don't have Performance Agent (used to be called Measureware) - you can check by trying the following command:

mwa status

If that returns a "not found" message, you don't have Performance Agent installed.

Assuming you don't, you have no historical data on what processes are consuming memory and how much, so you'll have to start from scratch.

What you probably need to do is look at the top X processes consuming memory over time, so you'll need some sort of cron job that gathers info like this:

ps -elf | awk '{printf "%6s %10s %8s %s\n",$4,$3,$10,$15}' | sort -rnk3 | head -20

which looks at the top 20 memory consumers over time.

From that you _might_ see which (if any) processes are consuming more memory.

Bare in mind though that the way memory management works, you'll never see the memory consumed by a process going down until there is actually memory pressure on the system... by which I mean that when processes allocate memory by calls to system calls like malloc, the process size will grow, but when a call to free is made for that memory you won't see the process size of the process shrink again unless there is actual memory pressure on the system...

HTH

Duncan

[Turgay Cavdar]

Hi,
kmeminfo shows units in pages so you have to multiply the values by page size (4KB).
Your buffer cache is nearly 1,3GB. Is it high? For oracle installs yes, you should consider to decrease dbc_max_pct between 5 and 10...

[Dennis Handly]

>swapinfo -a, shows very low activity.

(You should use "swapinfo -tam" and look at the total line.)

If you use top, can you see any memory growth in a process?
This would indicate either a leak or heap fragmentation.

[john korterman]

Hi Frank,

you could take a look at the memory consumption of individual processes, e.g.:

$ UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

which shows a per user sorted list.

Or a top ten of memory consumers:

$ UNIX95= ps -e -o ruser,sz,vsz,pid,ppid,args | grep -v '^RUSER' | sort -nr -k2,2 | head -10

If you execute this at regular intervals, you may see some development in the consumption.
But does your system actually stop because of insufficient memory?


regards,
John K.

[Bill Hassell]

You might have a process with a memory leak but you might also have a number of new programs running. A memory leak is a programming, probably something you cannot fix without contacting the manufacturer of the software. Also, you can only see a memory leak after you establish what is normal (after a reboot). The above ps commands will show you the biggest processes which is useful, especially if you see something you don't recognize that is using a lot of RAM. If all looks OK, then run the ps command into a log file so you can compare previous results.

[James R. Ferguson]

Hi Frank:

You note that "only recently have we had this". Is something or someone perchance killing (with 'kill -9') a process that uses shared memory? This would include 'fbackup' sessions. If so, the shared memory segments would be left lying around, consuming precious resources.

Regards!

...JRF...

[Don Morris_1]

That version of kmeminfo is beyond stale... it is simply an old script that was handed out with other Q4 tools. It pretty patently doesn't work [properly/completely] on your system -- so if you must use kernel intrusive tools, ask your support contact for a recent version. (I expect your kernel Dynamic data is non-zero and that's an important thing to know).

Getting back to the question at hand -- are you sure this is a leak in the first place? What exactly do you mean by "swapinfo shows PCT 60% or more"? If that's just reservation -- this is to be expected. You started Oracle, it presumably made some SGAs... but you didn't have client processes accessing it. As they start to -- they consume virtual address space, hence reserve swap... the load increases, the swap reservations increase... if the load decreases, the swap reservations will return.

Similarly -- HP-UX is *designed* to cache RAM for performance. The system will not be at all aggressive about reclaiming free memory in these caches (and there are several -- Buffer Cache, kernel dynamic memory arenas, etc.) when memory is above the lotsfree value (which is roughly 1%. 100Mb is about 2.44% of 4Gb, hence I wouldn't expect things to be reclaimed aggressively yet.

Doing a bit of math and trusting that that old kmeminfo isn't completely wrong on what it does recognize -- Total minus User minus DBC is 224560. Subtracting Static and Free -- that gives about 181251 pages for Kernel Dynamic or about 708Mb. Not unreasonable for a system under light pressure running Oracle.

So -- it looks like your big knob if you're really concerned is the buffer cache. And since that would naturally fill up as the system gets used, that may be your perceived leak in the first place -- you could set dbc_min_pct and dbc_max_pct to the same value to prevent buffer cache growth (it would prereserve the min and then just recycle those pages). As mentioned, if Oracle is doing direct I/O anyway, a smaller buffer cache is likely better for you anyway. At the least keeping the buffer cache static would give you some idea if this is your perceived leak, though I still would expect free memory to drop towards 1% and stay there in the common case.

[Laurent Menase]

If you want to use kmeminfo you should probably get a newer one.

The one you have looks to be an old one working on 11.0 only.....

but looks like you should reduce your buffer cache ( dbc_max_pct) . it is using 30% of the memory currently

[Frank de Vries]

Thanks , some good replies ,
I changed max_dbc_pct , which was 50%.
So I consider that a solution for now.

And I started to run UNIX95 at regualar intervals, so I wil see what comes out.

I didn't find the comments about kmeminfo not that relevant. To me Kmeminfo responded well to the kernel par change I did.

Tough allocating points on this one :)

[Dennis Handly]

>Bill: you can only see a memory leak after you establish what is normal (after a reboot).

The reboot is probably true for a kernel leak. But an application leak can be seen by restarting the application.
And using a leak detection tool like gdb.