High System or Kernel Memory Usage

[RahulS]

Hi,

I am observing high memory usage by system on almost all my HPUX11iv3 ( Sep 2009 release). Following are the output of commands for your reference.

This is Oracle 11g database server having SGA=8GB and PGA set to 1GB

# ./kmeminfo
tool: kmeminfo 9.04 - libp4 9.344 - libhpux 1.236 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "alonsum1"
core: /dev/kmem live
link: Sun Nov 08 10:10:23 WAT 2009
boot: Sun Feb 7 04:12:22 2010
time: Sat Feb 27 17:45:30 2010
nbpg: 4096 bytes


----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4186551 16.0g 100%
Free memory = 15364 60.0m 0%
User processes = 2455466 9.4g 59% details with -user
System = 1456230 5.6g 35%
Kernel = 1456204 5.6g 35% kernel text and data
Dynamic Arenas = 878794 3.4g 21% details with -arena
asyncdsk variab = 124855 487.7m 3%
spinlock_arena = 91111 355.9m 2%
reg_fixed_arena = 74216 289.9m 2%
vx_inode_kmcach = 65328 255.2m 2%
vx_global_kmcac = 63546 248.2m 2%
Other arenas = 459738 1.8g 11% details with -arena
Super page pool = 248191 969.5m 6% details with -kas
UAREA's = 14688 57.4m 0%
Static Tables = 316976 1.2g 8% details with -static
pfdat = 204421 798.5m 5%
inode = 52948 206.8m 1%
vhpt = 32768 128.0m 1%
ncache = 9495 37.1m 0%
text = 8990 35.1m 0% vmunix text section
Other tables = 8353 32.6m 0% details with -static
Buffer cache = 26 104.0k 0% details with -bufcache
UFC file mrg = 177190 692.1m 4%

# kcusage
Tunable Usage / Setting
=============================================
filecache_max 722509824 / 8154624000
maxdsiz 9764864 / 2147483647
maxdsiz_64bit 102367232 / 274877906944
maxfiles_lim 126 / 8192
maxssiz 106496 / 134217728
maxssiz_64bit 786432 / 1073741824
maxtsiz 7249920 / 1073741824
maxtsiz_64bit 318767104 / 8589934592
maxuprc 158 / 27001
max_thread_proc 101 / 3000
msgmni 2 / 30000
msgtql 0 / 30000
nflocks 17 / 30000
ninode 1298 / 242048
nkthread 926 / 52516
nproc 381 / 30000
npty 0 / 200
nstrpty 8 / 200
nstrtel 0 / 60
nswapdev 2 / 32
nswapfs 0 / 32
semmni 61 / 30000
semmns 3406 / 60000
shmmax 8657051648 / 4398046511104
shmmni 7 / 4096
shmseg 3 / 512

[Michael Steele_2]

What are you reporting for system? User? These numbers?

"...Free memory = 0%
User processes = 59%
System = 35%..."

Or are you concerned about what follows?

What is normal? What should you be seeing?

Attach these reports please:

a) glanc m memory usuage
b) UNIX96=1 ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15

[RahulS]

Thanks Michael,

Following are my concern

1. Free memory is 0%
2. System memory is 35% (5.6GB)

System memory usage seems to be on higher side ,

Sorry I cannot provide the glance output as we dont have it in our environment, and its controlled environment so need to time install any evaluation copy.

output of the command.

$ export UNIX96=1
$ ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15
ps: illegal option -- o
usage: ps [-edaxzflP] [-u ulist] [-g glist] [-p plist] [-t tlist] [-R prmgroup] [-Z psetidlist]
$ export UNIX95=1
$ ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15
328128 28444 1 S e000000316a23e40 ora_dbw4_crmdb1
328128 28442 1 S e000000316901e40 ora_dbw3_crmdb1
328128 28440 1 S e000000374d155c0 ora_dbw2_crmdb1
328128 28435 1 S e0000003169018c0 ora_dbw1_crmdb1
328128 28430 1 S e0000001fa137340 ora_dbw0_crmdb1
320256 29034 1 S e00000037614d940 ora_cjq0_crmdb1
320000 29207 1 S e0000003ad6ec272 oraclecrmdb1
320000 29080 1 S e00000019b12f9f2 oraclecrmdb1
320000 28472 1 S e00000037614d540 ora_mmon_crmdb1
320000 28453 1 S e0000003146ce940 ora_smon_crmdb1
320000 28093 1 S e000000316a23240 asm_gmon_+ASM1
320000 28085 1 S e00000037614d2c0 asm_rbal_+ASM1
320000 28079 1 S e000000374d15840 asm_lgwr_+ASM1
320000 28077 1 S e0000003291fc0c0 asm_dbw0_+ASM1
320000 22876 1 S e0000003d2e5c868 oraclecrmdb1

[Jeeshan]

what is your dbc_max_pct and dbc_min_pct value in kernel parameters?

[Dennis Handly]

>unix_shell: what is your dbc_max_pct and dbc_min_pct value in kernel parameters?

These are pretty useless since this is 11.31 and the vestigial buffer cache is only using .1 Mb.

[Emil Velez]

ITs probably all those oracle shared memory segments. too bad that you do not have glance and cannot "evaluate" it.

Do you have measureware and can you extract anything.

Since it is 11.31 you will need to look at filecache min and filecache max.

look at ipcs -ma and look at the size fields of your shared memory segments.

[WayneHP]

Lets see you have 16GB of memory.

8GB goes to Oracle SGA

and 1GB goes for each Oracle connection

you can have less than 8 connections because the rest of UNIX needs some memory, or paging/swapping begins. Are you sure you need a 1GB PGA?

PGA
Program Global Area. The PGA is a memory region containing data and control information for a single process (server or background). One PGA is allocated for each server process; the PGA is exclusive to that server process and is read and written only by Oracle code acting on behalf of that process. A PGA is allocated by Oracle when a user connects to an Oracle database and a session is created.

[WayneHP]

How many Oracle connections do you have?

http://download.oracle.com/docs/cd/B28359_01/server.111/b28274/memory.htm#i47865

This doc may help

[Michael Steele_2 · Accepted solution]

Hi

Based upon what you've provided I'd say your Oracle DBA has tuned oracle in this fashion and you will have to review ora_dbw, your largest consumer that appears to be behaving properly and is perfectly reasonable depending on the size of your buffercache.

For processes like the dbwr (which will touch much of the pages of your
buffercache most probably) this means that the reported used memory will
grow initially, and be reported as you see.

If you run 'ipcs -moba' and total up the SEGSZ you'll get a measure of how much total shared memory oracle is using.

From the HP-UX side, the related kernel parameters are 'dbc_min_pct' and 'dbc_max_pct'. Post them for comments and feedbacks.

From my point of view I would do nothing unless you are getting complaints from users. If you are then you have two reliable choices from the HP-UX side of the house, buy more RAM and / or reduce the number of process burdening the box.

Here is a good thread related to this problem. It discuss buffer cache and notes the importance of being properly patch in HP-UX 11.31.

So I guess I would start here, proper buffer cache patching for 11.31.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1193484

[Dennis Handly]

>Michael: the related kernel parameters are dbc_min_pct and dbc_max_pct.

These aren't so important for 11.31. And the file cache usage is only .7 Gb.