memory leak or not?

[mobidyc]

Hello,

on a B.11.11 server with 47Gb of memory installed, the kernel takes around 10Gb of memory ;(

# kmeminfo
tool: kmeminfo 5.19
unix: /stand/vmunix 11.11 64bit PA2.0 on "nr0u0151"
core: /dev/kmem live
link: Thu Sep 14 14:31:37 METDST 2006
boot: Sat Dec 2 11:46:33 2006
time: Thu Feb 1 16:10:21 2007
nbpg: 4096 bytes


----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 12573952 48.0g 100%
Free memory = 1027749 3.9g 8%
User processes = 7601841 29.0g 60% details with -user
System = 3917143 14.9g 31%
Kernel = 2659748 10.1g 21% kernel text and data
Dynamic Arenas = 1335977 5.1g 11% details with -arena
ALLOCB_MBLK_LM = 712882 2.7g 6%
M_TEMP = 220944 863.1m 2%
ALLOCB_MBLK_SM = 186043 726.7m 1%
ALLOCB_MBLK_MH = 88792 346.8m 1%
M_SPINLOCK = 25170 98.3m 0%
Other arenas = 102146 399.0m 1% details with -arena
Super page pool = 559881 2.1g 4% details with -kas
Static Tables = 636171 2.4g 5% details with -static
pfdat = 287757 1.1g 2%
htbl2_0 = 131072 512.0m 1%
nbuf = 106992 417.9m 1% bufcache headers
pfn_to_virt = 47959 187.3m 0%
bufhash = 40960 160.0m 0% bufcache hash headers
Other tables = 21430 83.7m 0% details with -static
Buffer cache = 1257395 4.8g 10% details with -bufcache

i've already looked for tunables parameters, the only who seems to be far too hight is dbc_max_pct (10%) but i believe it's not part of the kernel memory utilisation.
here are the last tunables modified:
STRMSGSZ 65535
dbc_max_pct 10
dnlc_hash_locks 512
max_thread_proc 1024
maxdsiz 0XFFFFF000
maxdsiz_64bit 21474836480
maxfiles 4096
maxfiles_lim 5120
maxssiz 0X17F00000
maxssiz_64bit 0X40000000
maxswapchunks 16384
maxtsiz 0x4000000
maxtsiz_64bit 0x40000000
maxuprc 2662
maxusers 1024
maxvgs 80
nfile (60*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
nflocks 3500
ninode (((NPROC+16+MAXUSERS)+32+(2*NPTY))*3)
nstrpty 60
semaem 32000
semmni 6656
semmns 25000
semmnu 3324
semume 64
shmmax 4294967295
shmmni 512
shmseg 128

have an idea?

thanks in advance.
Regards,
Cedrick Gaillard

[Steven E. Protter]

Shalom,

There are patches for OS related memory leaks and you could choose to pre-emptively install them.

Memory leaks show up on glance and even to as processes that continue to increase memory use and do not give back.

Commonly they are caused by poor programming techniques. Large vendors like Oracle provide patches for dealing with the issue in their applications and database server software.

Try to identify the process and then you may have a course of action.

SEP

[A. Clay Stephenson]

Your biggest chunk is the buffer cache and on 11.11 with 47GiB of memory, 10% is still way too high. 11.23 scales the buffer cache back down much faster and the buffer cache search routines are more efficient than with 11.11 or lower and thus 11.23 is better able to use very large buffer caches. 11.11, in almost all cases, tends to hit the "sweet spot" at something like 1600MiB. While you can't do this in 11.23 because it has been eliminated, if I were you in 11.11, I would go to a static buffer cache of 1600MiB by setting bufpages=409600. You could also do much the same thing by setting your dbc_max_pct and dbc_min_pct to about 3 or 4 but I prefer the static caches.

[Peter Godron]

Hi,
have you applied the latest patchsets ?

Your machine has only been running 2 months, I assume there were no earlier problems ?
Have applications been upgraded ?

As this is not the 11.23 of your earlier thread you may want to read:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=104309

Using glance to highlight problem processes may also help.

[mobidyc]

Hi all,

the only known story of this server is than a colleague have tried to Goldapps it one year ago wich result in softare incompatibility, an ignite recovery have been useful.

so, effectively, this server is not fully patched and we try to not reboot this system.

i'm going to open a call at HP's support.

if you have ideas between this time... ;)

PS: the buffer cache is not part of the dynamic arena of the kernel so, this parameter (to high i know) is not considered for the moment.

Regards,
Cedrick Gaillard

[Laurent Menase · Accepted solution]

Hi Cedrick


for that type of question, it looks reasonable to open a call to hp support. The answer is not so direct.
6% ALLOCB_MBLK_LM arena means that network stack is owning 6% of the memory. There are cases where it is due to a leak, or this can be a peak usage, or this can be also the normal activity if it is a system with a lot of cpu and a lot of NIC.

Regards,
Laurent Menase

[mobidyc]

thanks Laurent,

a call have been open to the HP call center, i'm waiting for their diagnostics.

if you have other explanations for the other areas please share it ;)

Cheers,
Cedrick Gaillard

[Don Morris_1]

What other explanations are you looking for?

+ The system is not under memory pressure (3.9Gb free), so the kernel memory policies favor caching for performance over releasing memory. Hence, memory used at peak is likely still around.

+ The super page pool is a kernel memory cache -- and only releases memory back to the system when all parts of a given super page are returned to it from the Arena layer. [And since you have a reasonable amount of memory in the Arena layer, the fact that there are uncoalesced super pages and hence memory hanging around in that cache isn't surprising.] On the plus side, new kernel allocations try this cache first (with some exceptions... there's always exceptions), so the kernel should stay out of your 3.9Gb until that's gone or it needs a larger page size than what's cached.

pfdats and translation entries are completely and utterly out of your control. This is the metadata the kernel needs to represent physical memory, its consumption scales directly with memory (more RAM, more metadata to represent it in the kernel/translate it in the kernel/etc.)

Buffer cache is at the max.. yes, but you told it it could go there (dbc_max_pct 10) and there isn't pressure forcing it down...

Frankly -- on first look, I don't see anything odd or wrong here... It looks to me like your typical NFS server style load (Streams clients tend to be networking... but there are others, so NFS is just my first guess). You're doing a lot of I/O that gets buffered in the Buffer Cache [so that grows up to the max], and then sending that across I/O adapters... so kernel metadata for the I/O itself gets created (streams arenas grow based on peak usage).. and your system is happy enough with the pressure that the garbage collection isn't particularly aggressive.

That's just first look, mind you -- so I have no problem with you having support check that the streams arenas aren't leaking instead of just reflecting a peak/current usage. Better safe than sorry... but I wouldn't be too worried about any of this until you hear back from them.

[mobidyc]

Thanks Don Morris,

these informations was what i've searched.

Regards,
Cedrick Gaillard