topic Re: High System or Kernel Memory Usage in Operating System - HP-UX

High System or Kernel Memory Usage

RahulS — Sat, 27 Feb 2010 14:59:18 GMT

Hi,

I am observing high memory usage by system on almost all my HPUX11iv3 ( Sep 2009 release). Following are the output of commands for your reference.

This is Oracle 11g database server having SGA=8GB and PGA set to 1GB

# ./kmeminfo
tool: kmeminfo 9.04 - libp4 9.344 - libhpux 1.236 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "alonsum1"
core: /dev/kmem live
link: Sun Nov 08 10:10:23 WAT 2009
boot: Sun Feb 7 04:12:22 2010
time: Sat Feb 27 17:45:30 2010
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4186551 16.0g 100%
Free memory = 15364 60.0m 0%
User processes = 2455466 9.4g 59% details with -user
System = 1456230 5.6g 35%
Kernel = 1456204 5.6g 35% kernel text and data
Dynamic Arenas = 878794 3.4g 21% details with -arena
asyncdsk variab = 124855 487.7m 3%
spinlock_arena = 91111 355.9m 2%
reg_fixed_arena = 74216 289.9m 2%
vx_inode_kmcach = 65328 255.2m 2%
vx_global_kmcac = 63546 248.2m 2%
Other arenas = 459738 1.8g 11% details with -arena
Super page pool = 248191 969.5m 6% details with -kas
UAREA's = 14688 57.4m 0%
Static Tables = 316976 1.2g 8% details with -static
pfdat = 204421 798.5m 5%
inode = 52948 206.8m 1%
vhpt = 32768 128.0m 1%
ncache = 9495 37.1m 0%
text = 8990 35.1m 0% vmunix text section
Other tables = 8353 32.6m 0% details with -static
Buffer cache = 26 104.0k 0% details with -bufcache
UFC file mrg = 177190 692.1m 4%

# kcusage
Tunable Usage / Setting
=============================================
filecache_max 722509824 / 8154624000
maxdsiz 9764864 / 2147483647
maxdsiz_64bit 102367232 / 274877906944
maxfiles_lim 126 / 8192
maxssiz 106496 / 134217728
maxssiz_64bit 786432 / 1073741824
maxtsiz 7249920 / 1073741824
maxtsiz_64bit 318767104 / 8589934592
maxuprc 158 / 27001
max_thread_proc 101 / 3000
msgmni 2 / 30000
msgtql 0 / 30000
nflocks 17 / 30000
ninode 1298 / 242048
nkthread 926 / 52516
nproc 381 / 30000
npty 0 / 200
nstrpty 8 / 200
nstrtel 0 / 60
nswapdev 2 / 32
nswapfs 0 / 32
semmni 61 / 30000
semmns 3406 / 60000
shmmax 8657051648 / 4398046511104
shmmni 7 / 4096
shmseg 3 / 512

Re: High System or Kernel Memory Usage

Michael Steele_2 — Sat, 27 Feb 2010 15:15:08 GMT

What are you reporting for system? User? These numbers?

"...Free memory = 0%
User processes = 59%
System = 35%..."

Or are you concerned about what follows?

What is normal? What should you be seeing?

Attach these reports please:

a) glanc m memory usuage
b) UNIX96=1 ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15

Re: High System or Kernel Memory Usage

RahulS — Sat, 27 Feb 2010 16:05:40 GMT

Thanks Michael,

Following are my concern

1. Free memory is 0%
2. System memory is 35% (5.6GB)

System memory usage seems to be on higher side ,

Sorry I cannot provide the glance output as we dont have it in our environment, and its controlled environment so need to time install any evaluation copy.

output of the command.

$ export UNIX96=1
$ ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15
ps: illegal option -- o
usage: ps [-edaxzflP] [-u ulist] [-g glist] [-p plist] [-t tlist] [-R prmgroup] [-Z psetidlist]
$ export UNIX95=1
$ ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15
328128 28444 1 S e000000316a23e40 ora_dbw4_crmdb1
328128 28442 1 S e000000316901e40 ora_dbw3_crmdb1
328128 28440 1 S e000000374d155c0 ora_dbw2_crmdb1
328128 28435 1 S e0000003169018c0 ora_dbw1_crmdb1
328128 28430 1 S e0000001fa137340 ora_dbw0_crmdb1
320256 29034 1 S e00000037614d940 ora_cjq0_crmdb1
320000 29207 1 S e0000003ad6ec272 oraclecrmdb1
320000 29080 1 S e00000019b12f9f2 oraclecrmdb1
320000 28472 1 S e00000037614d540 ora_mmon_crmdb1
320000 28453 1 S e0000003146ce940 ora_smon_crmdb1
320000 28093 1 S e000000316a23240 asm_gmon_+ASM1
320000 28085 1 S e00000037614d2c0 asm_rbal_+ASM1
320000 28079 1 S e000000374d15840 asm_lgwr_+ASM1
320000 28077 1 S e0000003291fc0c0 asm_dbw0_+ASM1
320000 22876 1 S e0000003d2e5c868 oraclecrmdb1

Re: High System or Kernel Memory Usage

Jeeshan — Sun, 28 Feb 2010 04:32:51 GMT

what is your dbc_max_pct and dbc_min_pct value in kernel parameters?

Re: High System or Kernel Memory Usage

Dennis Handly — Sun, 02 Oct 2011 02:28:44 GMT

>unix_shell: what is your dbc_max_pct and dbc_min_pct value in kernel parameters?

These are pretty useless since this is 11.31 and the vestigial buffer cache is only using .1 Mb.

Re: High System or Kernel Memory Usage

Emil Velez — Sun, 28 Feb 2010 04:46:13 GMT

ITs probably all those oracle shared memory segments. too bad that you do not have glance and cannot "evaluate" it.

Do you have measureware and can you extract anything.

Since it is 11.31 you will need to look at filecache min and filecache max.

look at ipcs -ma and look at the size fields of your shared memory segments.

Re: High System or Kernel Memory Usage

WayneHP — Sun, 28 Feb 2010 14:44:12 GMT

Lets see you have 16GB of memory.

8GB goes to Oracle SGA

and 1GB goes for each Oracle connection

you can have less than 8 connections because the rest of UNIX needs some memory, or paging/swapping begins. Are you sure you need a 1GB PGA?

PGA
Program Global Area. The PGA is a memory region containing data and control information for a single process (server or background). One PGA is allocated for each server process; the PGA is exclusive to that server process and is read and written only by Oracle code acting on behalf of that process. A PGA is allocated by Oracle when a user connects to an Oracle database and a session is created.

Re: High System or Kernel Memory Usage

WayneHP — Sun, 28 Feb 2010 15:45:40 GMT

How many Oracle connections do you have?

http://download.oracle.com/docs/cd/B28359_01/server.111/b28274/memory.htm#i47865

This doc may help

Re: High System or Kernel Memory Usage

Michael Steele_2 — Sun, 28 Feb 2010 20:17:34 GMT

Hi

Based upon what you've provided I'd say your Oracle DBA has tuned oracle in this fashion and you will have to review ora_dbw, your largest consumer that appears to be behaving properly and is perfectly reasonable depending on the size of your buffercache.

For processes like the dbwr (which will touch much of the pages of your
buffercache most probably) this means that the reported used memory will
grow initially, and be reported as you see.

If you run 'ipcs -moba' and total up the SEGSZ you'll get a measure of how much total shared memory oracle is using.

From the HP-UX side, the related kernel parameters are 'dbc_min_pct' and 'dbc_max_pct'. Post them for comments and feedbacks.

From my point of view I would do nothing unless you are getting complaints from users. If you are then you have two reliable choices from the HP-UX side of the house, buy more RAM and / or reduce the number of process burdening the box.

Here is a good thread related to this problem. It discuss buffer cache and notes the importance of being properly patch in HP-UX 11.31.

So I guess I would start here, proper buffer cache patching for 11.31.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1193484

Re: High System or Kernel Memory Usage

Dennis Handly — Sun, 28 Feb 2010 22:59:50 GMT

>Michael: the related kernel parameters are dbc_min_pct and dbc_max_pct.

These aren't so important for 11.31. And the file cache usage is only .7 Gb.

Re: High System or Kernel Memory Usage

Michael Steele_2 — Sun, 28 Feb 2010 23:26:12 GMT

Oracle maintains its own buffer cache inside the system global area (SGA) for each instance.

http://www.dbspecialists.com/files/presentations/buffercache.html

Re: High System or Kernel Memory Usage

RahulS — Sun, 28 Feb 2010 23:32:11 GMT

Please the below filecache_max usage pattern of my two Oracle 11g RAC database servers.

Day Date Time "
(BRM DB Node1)" "
(BRM DB node2)"
Bytes % Bytes %
Sat 2/27/2010 23:00 1803579392 11 2958098432 18.1
Sun 2/28/2010 0:00 1938280448 11.9 16322625536 100
Sun 2/28/2010 1:00 1905328128 11.7 16016777216 98.1
Sun 2/28/2010 2:00 1918468096 11.8 15594962944 95.5
Sun 2/28/2010 3:00 1922412544 11.8 15480061952 94.8
Sun 2/28/2010 4:00 1971060736 12.1 15217262592 93.2
Sun 2/28/2010 5:00 1868517376 11.4 15106449408 92.5
Sun 2/28/2010 6:00 1879154688 11.5 15014379520 92
Sun 2/28/2010 7:00 1888792576 11.6 14519771136 88.9
Sun 2/28/2010 8:00 1893126144 11.6 14189719552 86.9
Sun 2/28/2010 9:00 1906495488 11.7 14113128448 86.5
Sun 2/28/2010 10:00 1919987712 11.8 14014951424 85.8
Sun 2/28/2010 11:00 1936580608 11.9 13673652224 83.8
Sun 2/28/2010 12:00 1934462976 11.8 13445591040 82.4
Sun 2/28/2010 13:00 1950392320 11.9 13200674816 80.9
Sun 2/28/2010 14:00 1963429888 12 12804304896 78.4
Sun 2/28/2010 15:00 1972707328 12.1 12385775616 75.9
Sun 2/28/2010 16:00 1984831488 12.2 11934097408 73.1
Sun 2/28/2010 17:00 2005561344 12.3 11391574016 69.8
Sun 2/28/2010 18:00 2022236160 12.4 11014860800 67.5
Sun 2/28/2010 19:00 2033881088 12.5 10861117440 66.5
Sun 2/28/2010 20:00 2042576896 12.5 10673901568 65.4
Sun 2/28/2010 21:00 1968144384 12.1 10589315072 64.9
Sun 2/28/2010 22:00 1981603840 12.1 10480779264 64.2

Re: High System or Kernel Memory Usage

RahulS — Sun, 28 Feb 2010 23:35:45 GMT

Please find the attached formatted usage pattern

Re: High System or Kernel Memory Usage

Michael Steele_2 — Mon, 01 Mar 2010 00:19:30 GMT

Hi

The arbiter here is going to be buffer cache hits, which is expected to be a 90 to 100% hit ratio.

I can see from your last posting that something changed, you dropped from 100% to 60%. How did you do this? Kill processes? Wait for a big sql to complete?

Review the link I post above, I'm interested in seeing this report:

SELECT 250 * TRUNC (rownum / 250) + 1 || ' to ' ||
250 * (TRUNC (rownum / 250) + 1) "Interval",
SUM (count) "Buffer Cache Hits"
FROM v$recent_bucket
GROUP BY TRUNC (rownum / 250)

Interval Buffer Cache Hits
--------------- --------------------
1 to 250 16083
251 to 500 11422
501 to 750 683
751 to 1000 177

Re: High System or Kernel Memory Usage

Michael Steele_2 — Mon, 01 Mar 2010 00:26:12 GMT

This is interesting, contradictory, and has some good arguements.

http://raj_oracle90.tripod.com/sitebuildercontent/sitebuilderfiles/whya99percentbuffercacheratioisnotok-carymillsap.pdf

Re: High System or Kernel Memory Usage

Horia Chirculescu — Mon, 01 Mar 2010 08:34:33 GMT

Hello,

>maxdsiz 9764864 / 2147483647
>maxdsiz_64bit 102367232 / 274877906944

274 877 906 944 = ~270G This is huge. Are you sure you want this?

>semmni 61 / 30000
>semmns 3406 / 60000

The number of semaphores are also very high (30000 !)

>shmmax 8657051648 / 4398046511104

4 398 046 511 104 = ~ 4T Also, some huge value!

How does it look like the output for:

ipcs -moba

Oracle should use only one segment - and the biggest one from your system (maybe) - the one with 8657051648 bytes.

>shmmni 7 / 4096
>shmseg 3 / 512

Why did you set a very large segment identifiers? - 4096 is a huge value!

Horia.

Re: High System or Kernel Memory Usage

RahulS — Mon, 01 Mar 2010 11:40:28 GMT

Hi,

All the parameters below are default.

maxdsiz 9764864 / 2147483647
maxdsiz_64bit 102367232 / 274877906944
maxfiles_lim 126 / 8192
maxssiz 106496 / 134217728
maxssiz_64bit 786432 / 1073741824
maxtsiz 7249920 / 1073741824

Please see the attached document for ipcs -moba output. SEGSZ for oracle is 4GB

Re: High System or Kernel Memory Usage

Michael Steele_2 — Mon, 01 Mar 2010 12:33:05 GMT

T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME

m 65542 0xc3eb8a14 --rw-rw---- oracle dba oracle dba 1428 4311756800 18386 20019 14:26:38

NATTCH 1428
SEGSZ 4311756800

Number attached: 1428, that is processes - You are pigging out this system

SEGMENT SIZE - In G bytes, 4 GB. Which is fine.

You can figure out the rest for total shared memory consumption yourself. However, you've got to tune down the number of users or processes on this box or buy more RAM.

Re: High System or Kernel Memory Usage

Horia Chirculescu — Mon, 01 Mar 2010 13:04:27 GMT

Can you please share with us the output of

ps -e |grep oracle |wc -l

Horia.

Re: High System or Kernel Memory Usage

RahulS — Mon, 01 Mar 2010 14:07:07 GMT

Here it is
--------------------------
DB Node1

# ps -aef|grep oracle|wc -l
764
--------------------------
DB Node2
$ ps -aef|grep oracle |wc -l
126