kernel is using most of memory

[Tero Palonen]

Hi,

i have a strange problem, my server K580 hpux 11.11 has memory 1.5 Gb but system is using most of it. After boot memory usage is immediately more than 98% and no applications is running. I wonder what can i do to release more memory for applications.

dbc_max_pct 5 dbc_min_pct 2

Glance

Total VM : 203mb Sys Mem : 1.4gb User Mem: 72mb Phys Mem: 1.5gb
Active VM: 138mb Buf Cache: 35mb Free Mem: 17mb

# vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 29194 1025 58 29 15 2 2 0 18 1369 4872 250 2 3 94
1 1 0 29194 1053 1 0 12 32 24 0 49 1960 4208 293 0 3 97
1 1 0 29194 1037 0 0 10 25 19 0 39 1952 4248 285 1 5 94
1 1 0 29194 1016 0 0 10 20 16 0 36 1988 5051 321 2 6 92
2 0 0 30043 1008 0 0 8 32 20 0 36 2038 6022 358 1 7 92

# ./freemem
freemem= 24625 gpgslim= 1024 lotsfree= 5053 desfree= 1024 bufpages= 12955 dbcminpgs= 7864 dbcmaxpgs= 19661 minfree= 256

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1536 52 1484 3% 0 - 1 /dev/vg00/lvol2
dev 1536 0 1536 0% 0 - 2 /dev/vg01/lvol2
reserve - 113 -113
total 3072 165 2907 5% - 0 -

BR/ Tero

[Don Morris_1 · Accepted solution]

More likely than not you've got some tunables set too high (the SysV IPC semaphore tunables can easily get you in this state... if you're running a version before 11.23.0409 you can eat a lot with setting max_mem_window too high [on 11.11 that interacts with shmmni].

Get the unsupported kmeminfo tool from:

ftp://eh:spear9@hprc.external.hp.com/

Run it using kmeminfo /stand/vmunix /dev/kmem and report the results.

[Bill Hassell]

You've got a massively large kernel parameter (or two or three...). Paste a copy of your /stand/system file starting with the first changed parameter at the bottom of the list (typically, it will start with the dbc_xxx values).

[TwoProc]

Yeah, that's a lot of ram used by sys. I just got the same glance data from one of our old D's with 3G ram, and it is only using 160Mb for sys - whereas yours is chewing up 1.4G.

My first thought was that it was dbc_max_pct - but that's not it - must be something else in the kernel, or a memory leak of a system program.

Can you post to us the output of "kmtune" ?

Also, since you've got glance, run the graphics version of "gpm" and go to "reports" and look at "process list". Once it is up, go to "Configure" and then "Filter" and click on "Disable All Filters", then close the dialog box. You should then be able to sort on Resident Memory usage by clicking on "Configure" , then "Sort Fields", then dragging the Res Mem columns all the way to the left and hitting "Done".

This should list you the highest memory consuming processes in the box, and give you clue of where to start looking.

[Tero Palonen]

Hi,

I put attachment of kernel parameters.
This is results of kmeminfo program:


# ./kmeminfo /stand/vmunix /dev/kmem
kmeminfo (5.01)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Fri Aug 26 13:33:16 EETDST 2005
boot: Fri Aug 26 13:47:15 2005
dump: Fri Aug 26 16:04:47 2005
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 393212 1.5g 100%
Free memory = 13898 54.3m 4%
User processes = 12408 48.5m 3% details with -user
System = 366003 1.4g 93%
Kernel = 353646 1.3g 90% kernel text and data
Dynamic Arenas = 317260 1.2g 81% details with -arena
M_SYSVMSGB = 262129 1023.9m 67%
M_TEMP = 41332 161.5m 11%
M_SPINLOCK = 3537 13.8m 1%
M_SWAP = 1564 6.1m 0%
M_VXVM = 1368 5.3m 0%
Other arenas = 7330 28.6m 2% details with -arena
Super page pool = 4532 17.7m 1% details with -kas
Static Tables = 24888 97.2m 6% details with -static
pfdat = 8848 34.6m 2%
htbl2_0 = 4096 16.0m 1%
text = 2649 10.3m 1% vmunix text section
nbuf = 2416 9.4m 1% bufcache headers
pfn_to_virt = 1474 5.8m 0%
Other tables = 5403 21.1m 1% details with -static
Buffer cache = 12357 48.3m 3% details with -bufcache
#

BR/Tero

[Don Morris_1]

Yup... your SysV message buffer arena is just huge compared to your total memory. Drop your msg* tunables, because they're too big for your system. msgseg looks like the most likely culprit... the kernel's going to allocate msgseg * msgsiz worth of space from that arena on boot.

[TwoProc]

Yup, Don nailed it down for you.

Mine look like the following on my D box:
msgmap 8192 - (MSGTQL+2)
msgmax 32768 Y 32768
msgmnb 65535 Y 65535
msgmni 819 - (NPROC)
msgseg 32760 - (MSGTQL*4)
msgssz 128 - 128
msgtql 8190 - (NPROC*10)

Yours would of course vary, but this can give you a bit of a guideline to see what is way out there.

[Tero Palonen]

Hi,

now Glance shows sys mem 290 Mb and memory usage about 32% on "empty" machine.
I did what Don Morris told me and changed 7 kernel parameters (msgmap - msgtql) to smaller ones.

Thanks for you all, now i can leave office happily and enjoy weekend.

# ./kmeminfo /stand/vmunix /dev/kmem
kmeminfo (5.01)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Fri Aug 26 16:58:53 EETDST 2005
boot: Fri Aug 26 17:16:03 2005
dump: Fri Aug 26 17:23:37 2005
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 393212 1.5g 100%
Free memory = 270695 1.0g 69%
User processes = 31275 122.2m 8% details with -user
System = 88951 347.5m 23%
Kernel = 77028 300.9m 20% kernel text and data
Dynamic Arenas = 18578 72.6m 5% details with -arena
M_TEMP = 7275 28.4m 2%
M_SWAP = 1566 6.1m 0%
M_VXVM = 1368 5.3m 0%
M_LVM = 1322 5.2m 0%
M_IOSYS = 856 3.3m 0%
Other arenas = 6191 24.2m 2% details with -arena
Super page pool = 28479 111.2m 7% details with -kas
Static Tables = 22935 89.6m 6% details with -static
pfdat = 8858 34.6m 2%
htbl2_0 = 4096 16.0m 1%
text = 2649 10.3m 1% vmunix text section
pfn_to_virt = 1476 5.8m 0%
bufhash = 1280 5.0m 0% bufcache hash headers
Other tables = 4575 17.9m 1% details with -static
Buffer cache = 11923 46.6m 3% details with -bufcache
#
Glance

Total VM : 188mb Sys Mem : 290mb User Mem: 159mb Phys Mem: 1.5gb
Active VM: 132mb Buf Cache: 47mb Free Mem: 1.0gb

BR/ Tero

[Victor_101]

hi,
can anyone help me out with my output,

root@Kddrac1/$>./kmeminfo
kmeminfo (5.01)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Fri Jun 17 10:50:12 GMT 2005
boot: Fri Jul 29 09:58:35 2005
dump: Mon Sep 26 15:05:09 2005
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 2097152 8.0g 100%
Free memory = 17109 66.8m 1%
User processes = 1523420 5.8g 73% details with -user
System = 551539 2.1g 26%
Kernel = 402685 1.5g 19% kernel text and data
Dynamic Arenas = 229794 897.6m 11% details with -arena
M_TEMP = 113097 441.8m 5%
ALLOCB_MBLK_LM = 37808 147.7m 2%
M_SPINLOCK = 14149 55.3m 1%
M_DEVBUF = 10353 40.4m 0%
M_NFS_GEN = 9939 38.8m 0%
Other arenas = 44448 173.6m 2% details with -arena
Super page pool = 27901 109.0m 1% details with -kas
Static Tables = 118535 463.0m 6% details with -static
pfdat = 47938 187.3m 2%
nbuf = 30992 121.1m 1% bufcache headers
htbl2_0 = 16384 64.0m 1%
pfn_to_virt = 7989 31.2m 0%
bufhash = 5120 20.0m 0% bufcache hash headers
Other tables = 10111 39.5m 0% details with -static
Buffer cache = 148854 581.5m 7% details with -bufcache


am running oracle 9i database with 11i E-business suite on a clustered environment (MCSG) with an EVA 3000..
the above details are fr the primary DB node where 4.5GB has been assigned to Oracle..

[Bill Hassell]

For Victor, from kmeminfo:

> Physical memory = 2097152 8.0g 100%
> Free memory = 17109 66.8m 1%
> User processes = 1523420 5.8g 73% details with -user
> System = 551539 2.1g 26%
> Kernel = 402685 1.5g 19% kernel text and data

Yes, you have one or more kernel parameters that are far too large. But kmtune just tells us that the kernel is massively large (2+ Gb), but not why. You need to show the bottom portion of your /system file where the changed parameters are located.