General
cancel
Showing results for 
Search instead for 
Did you mean: 

HPUX 11.31 Kernel Memory usage is high

SOLVED
Go to solution
Mark Fisher_4
Frequent Advisor

HPUX 11.31 Kernel Memory usage is high

We are now building a lot of new 11.31 servers and have noticed that the kernel memory useage is quite large compared to 11.23. For example we have a rx3600 with 4GB of physical memory and 2GB of the memory is consumed by kernel (or System Memory as view from glance).

I ran a kmeminfo and determined that the kernel memory consumption is from:

Dynamic Arenas = 149759 585.0m 14% details with -arena

Super page pool = 195074 762.0m 19% details with -kas

Static Tables = 77723 303.6m 7% details with -static

I then ran a simple malloc program to allocate memory and put the system under memory pressure. After all the free memory was exhausted, we started swaping, which was surprisingly very slow. I continue to put the system under memory pressure untill we run out of swap.

What is surprising is the kernel memory never shrank. There was no memory reclaimation from the kernel.

I expected the Super Page Pool to get trimmed back and free up PFN's and provide free memory but it never did. And whats amazing is the number of "free" entries in all the Superpage polls was never reduced. A program is mallocing memory and the kernel refuses to trim down all the free buckets in the Superpage pool, which is a waste.

My opinion is this is a major problem with 11.31 and the Super page pool management does not work. Would anyone like to comment on this?

Following is the kmeminfo for the Superpage pool. Notice the huge number of free entries that just consume memory and never get used.

Kernel memory Allocation Superpage pool (KAS):

total_dynamic_pages 342847

sp_pool_t 0xe0000001050ae580
sp_total_pages = 103008
sp_num_superpages = 75
sp_free_pages = 43030

Size sp_pool_head_t count free highest spl_next
0 4KB 0xe0000001050ae5f0 0 136 140 0xe000000126fef000
1 8KB 0xe0000001050ae618 0 15 99 0xe000000170fe6000
2 16KB 0xe0000001050ae640 0 66 1007 0xe00000013c7d8000
3 32KB 0xe0000001050ae668 0 261 659 0xe000000145dc8000
4 64KB 0xe0000001050ae690 6 486 515 0xe00000017eec0000
5 128KB 0xe0000001050ae6b8 0 301 365 0xe000000169140000
6 256KB 0xe0000001050ae6e0 4 159 171 0xe000000165ac0000
7 512KB 0xe0000001050ae708 0 67 71 0xe00000017e380000
8 1MB 0xe0000001050ae730 37 13 22 0xe000000173e00000
9 2MB 0xe0000001050ae758 0 2 6 0xe000000163e00000
10 4MB 0xe0000001050ae780 7 0 1 0xe0000001050ae780
11 8MB 0xe0000001050ae7a8 0 0 1 0xe0000001050ae7a8
12 16MB 0xe0000001050ae7d0 21 0 0 0xe0000001050ae7d0

sp_pool_t 0xe0000001050ae080
sp_total_pages = 227010
sp_num_superpages = 10818
sp_free_pages = 155888

Size sp_pool_head_t count free highest spl_next
0 4KB 0xe0000001050ae0f0 0 23906 47128 0xe00000015fcc2000
1 8KB 0xe0000001050ae118 2080 20711 26299 0xe0000001a8736000
2 16KB 0xe0000001050ae140 7575 9054 9235 0xe000000167710000
3 32KB 0xe0000001050ae168 0 4097 4130 0xe000000194758000
4 64KB 0xe0000001050ae190 723 1080 1530 0xe000000180190000
5 128KB 0xe0000001050ae1b8 0 122 381 0xe000000184fc0000
6 256KB 0xe0000001050ae1e0 232 6 44 0xe00000017dbc0000
7 512KB 0xe0000001050ae208 0 0 2 0xe0000001050ae208
8 1MB 0xe0000001050ae230 166 0 1 0xe0000001050ae230
9 2MB 0xe0000001050ae258 0 0 1 0xe0000001050ae258
10 4MB 0xe0000001050ae280 16 0 1 0xe0000001050ae280
11 8MB 0xe0000001050ae2a8 0 0 1 0xe0000001050ae2a8
12 16MB 0xe0000001050ae2d0 26 0 0 0xe0000001050ae2d0

Total number of free page on pools: 198918
5 REPLIES
Steven E. Protter
Exalted Contributor

Re: HPUX 11.31 Kernel Memory usage is high

Shalom,

check vx_ninode settings.

This defaults to zero which lets the system set it high and use lots of resources.

Check the size of the file cache.

filecache_max 7346111692 15% Imm (auto disabled)
filecache_min 2448703897 5% Imm (auto disabled)


Lowering that max figure can save you a ton of memory.

Also, though ninode is supposedly for HFS only, it defaults to a huge number and the system (11.31) generally uses less than 1000.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Don Morris_1
Honored Contributor

Re: HPUX 11.31 Kernel Memory usage is high

So... for future reference, if you're going to cite kernel arena consumption, *which* arenas [kmeminfo will show the top 5 by default] is pretty much always interesting/important since it gives an idea of what the workload is.

The short version: Yeah, likely this is vx_ninode causing over-aggressive caching again. Seriously consider tuning it away from the aggressive defaults. More information at:
http://docs.hp.com/en/5992-5741/apbs02.html
or
http://docs.hp.com/en/5900-0032/ch01s05.html
depending on VxFS version.

Longer version: The super page pool is almost certainly working as designed (I have never, ever seen it fail to free a page which is actually free -- the last part is the kicker, though). The problem you're seeing is that the little pieces of the pages are still out in use in the Arenas above the SPP layer [SPP is a large translation cache for Arenas]. Until those pieces come back, the SPP does not release memory.

The arena pieces in v3 are usually stuck (this is where I have to guess a bit) because the Filesystem holds on to inodes -- and the inodes in v3 have other file cache metadata affiliated (this will show as the Region arena, misc region arena, spinlock arena, etc.). Until the inode is uncached... that metadata doesn't come back -- hence vx_ninode tuning reduces the extra caching.

Versions of VxFS, the vm module of the kernel are also relevant here (later VxFS/VM has introduced a memory reclamation request from VM to VxFS to try to get this shrunk more aggressively, there are vhand fixes, etc. etc.)

And yes -- the super page pool in v2 operated the exact same way [well, v3 actually does a better job re-using pieces in the allocation paths]... the difference is entirely in the in-use Arena pieces.

Mark Fisher_4
Frequent Advisor

Re: HPUX 11.31 Kernel Memory usage is high

I don't think this is inode cache issue. From kmeminfo it looks like its only consuming 7 MB of memory. Let me provide the kmeminfo summary output here and you can comment on what seems to be consuming the kernel memory.

Physical memory = 1039395 4.0g 100%
Free memory = 441320 1.7g 42%
User processes = 0 0.0 0% details with -user
System = 548 2.1m 0%
Kernel = 0 0.0 0% kernel text and data
Dynamic Arenas = 149759 585.0m 14% details with -arena
vx_global_kmcac = 17260 67.4m 2%
spinlock_arena = 16614 64.9m 2%
ALLOCB_MBLK_SM = 12782 49.9m 1%
M_DYNAMIC = 8678 33.9m 1%
ALLOCB_MBLK_DA = 8133 31.8m 1%
Other arenas = 86292 337.1m 8% details with -arena
Super page pool = 195074 762.0m 19% details with -kas
Static Tables = 77723 303.6m 7% details with -static
pfdat = 50751 198.2m 5%
text = 9069 35.4m 1% vmunix text section
vhpt = 8192 32.0m 1%
bss = 5539 21.6m 1% vmunix bss section
inode = 1792 7.0m 0%
Other tables = 2378 9.3m 0% details with -static
Buffer cache = 270 1.1m 0% details with -bufcache
UFC meta mrg = 278 1.1m 0%
UFC file mrg = 20673 80.8m 2%
Don Morris_1
Honored Contributor
Solution

Re: HPUX 11.31 Kernel Memory usage is high

Side thought -- what version of kmeminfo is that? (The User sum of 0 and the kernel total of 0 makes me wonder why it isn't adding things up right).

Back to the problem -- ok, so you've got fragmentation of the super pages, and besides VxFS network buffers are the next big thing. Both of those clients do some caching at their level... there's a change in the 0909 patch that may be playing here, what's the output of:

what /stand/vmunix | grep vm_kmem

?

If it isn't B11.31.0909LR (or PHKL_40240) you should probably pick up PHKL_39401 or PHKL_40240 as you see fit.
Mark Fisher_4
Frequent Advisor

Re: HPUX 11.31 Kernel Memory usage is high

Looks like were running an older version of that patch:

PHKL_39747

Here's the output from your requested command:

$Revision: vm_kmem: @(#) em_64-perf R11.31_HOT_BL2009_0417_1 PATCH_11.31 PHKL_39747