memory leaking SLES 9 SP3 x86_64, Oracle iAS 10gR3

[Liviu I.]

Hi,

The problem relates to a server running SLES 9 SP3 (x86_64) with Oracle Application Server 10g R3.

The memory (16GB) gets filled up constantly, and when swapping the performance penalty is very large. What I cannot figure out is where does the memory get used because when the OAS is stopped, and all the HP server management software, etc. I have the following:

:~ # ps -e -o 'vsz pid ruser cpu time args' |sort -nr
VSZ PID RUSER CPU TIME COMMAND
52924 30119 root - 00:00:00 sort -nr
40252 13326 liviu - 00:00:00 sshd: liviu@pts/0
40104 13321 root - 00:00:00 sshd: liviu [priv]
26804 17088 root - 00:00:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
20040 13346 root - 00:00:00 su -
13864 13327 liviu - 00:00:00 -bash
13448 11153 ntp - 00:00:00 /usr/sbin/ntpd -p /var/lib/ntp/var/run/ntp/ntpd.pid -u ntp -i /var/lib/ntp
12692 13347 root - 00:00:00 -bash
7808 30118 root - 00:00:00 ps -e -o vsz pid ruser cpu time args
7012 10775 root - 00:00:00 /sbin/mingetty tty6
7012 10774 root - 00:00:00 /sbin/mingetty tty5
7012 10773 root - 00:00:00 /sbin/mingetty tty4
7012 10772 root - 00:00:00 /sbin/mingetty tty3
7012 10771 root - 00:00:00 /sbin/mingetty tty2
7012 10770 root - 00:00:00 /sbin/mingetty --noclear tty1
6808 9497 root - 00:00:00 /usr/sbin/cron
6652 7342 root - 00:00:00 /usr/sbin/hwinfo
3572 5237 root - 00:00:00 /sbin/syslogd -a /var/lib/ntp/dev/log
3536 5528 root - 00:00:00 /sbin/resmgrd
2744 5255 root - 00:00:00 /sbin/klogd -c 1 -2 -x
2560 6255 root - 00:00:00 [hwscand]
2492 7341 root - 00:00:00 cut -c 1-250
2484 7340 root - 00:00:00 expand
2480 3268 root - 00:00:00 /usr/sbin/irqbalance
656 1 root - 00:00:05 init [3]
0 10710 root - 00:00:00 [pdflush]
0 10703 root - 00:00:03 [pdflush]
0 4789 root - 00:00:00 [khubd]
0 2078 root - 00:00:00 [reiserfs/3]
0 2077 root - 00:00:00 [reiserfs/2]
0 2076 root - 00:00:00 [reiserfs/1]
0 2075 root - 00:00:00 [reiserfs/0]
0 1397 root - 00:00:00 [kseriod]
0 162 root - 00:00:00 [aio/3]
0 161 root - 00:00:00 [aio/2]
0 160 root - 00:00:00 [aio/1]
0 159 root - 00:00:00 [aio/0]
0 158 root - 00:01:13 [kswapd0]
0 131 root - 00:00:00 [kblockd/3]
0 130 root - 00:00:00 [kblockd/2]
0 129 root - 00:00:00 [kblockd/1]
0 128 root - 00:00:00 [kblockd/0]
0 15 root - 00:00:00 [kacpid]
0 14 root - 00:01:25 [khelper]
0 13 root - 00:00:00 [events/3]
0 12 root - 00:00:00 [events/2]
0 11 root - 00:00:00 [events/1]
0 10 root - 00:00:00 [events/0]
0 9 root - 00:00:00 [ksoftirqd/3]
0 8 root - 00:00:06 [migration/3]
0 7 root - 00:00:00 [ksoftirqd/2]
0 6 root - 00:00:06 [migration/2]
0 5 root - 00:00:00 [ksoftirqd/1]
0 4 root - 00:00:12 [migration/1]
0 3 root - 00:00:00 [ksoftirqd/0]
0 2 root - 00:00:09 [migration/0]

:~ # free
total used free shared buffers cached
Mem: 16410344 16307224 103120 0 171448 5934236
-/+ buffers/cache: 10201540 6208804
Swap: 4198312 4404 4193908

:~ # cat /proc/meminfo
MemTotal: 16410344 kB
MemFree: 103120 kB
Buffers: 171448 kB
Cached: 5934236 kB
SwapCached: 1664 kB
Active: 34300 kB
Inactive: 6075204 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16410344 kB
LowFree: 103120 kB
SwapTotal: 4198312 kB
SwapFree: 4193908 kB
Dirty: 800 kB
Writeback: 0 kB
Mapped: 9716 kB
Slab: 191932 kB
Committed_AS: 40577820 kB
PageTables: 808 kB
VmallocTotal: 536870911 kB
VmallocUsed: 43480 kB
VmallocChunk: 536827423 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

So I get that bytes 10201540 (~10GB) are used on the system when the applications are stopped. I suspect a driver/kernel memory leak. Am I right ? What tools/procedure any of you know to investigate further this.

Thank you
Liviu.

[Steven E. Protter]

Shalom,

I am betting based on experience that its Oracle.

Here is a tool to help.

http://www.hpux.ws

go to the memory leak tool.

As soon as I kick the server in the teeth and bring up the website.

Sorry for the lack of a direct link.

SEP

[Liviu I.]

Hi Steven,

your script does pretty much the same as the first command (please bear in mind, that I think you have to change the header : Vmem and Resident Memory).

I cannot understand where is the memory leaking, because I suspected oracle at first, but why is the memory used (10 GB) even when the oracle is closed.

Thank you for any suggestion.
Liviu.

[Liviu I.]

Just for reference:

After upgrading only the kernel to : 2.6.5-7.311-smp (part of SLES9-SP4) the problem seems to be solved.

Is this memory leaking in the SP3 kernel ?

Liviu.

[Steven E. Protter]

Shalom,

The purpose of the script, http://www.hpux.ws/?p=8 is to identify the process that uses more memory over time, not diagnose the problem.

The problem is almost always bad application code.

SEP

[Liviu I.]

SLES 9 - SP4 plus online updates, solved the question.

L.