Sys Mem consuming > 1/3 of Physical RAM

[Rajesh K Rajan]

KCTUNE -S :
Tunable Value Expression Changes
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
ksi_alloc_max 65536 nproc*8 Immed
max_thread_proc 1200 1200 Immed
maxfiles 20480 20480
maxfiles_lim 20480 20480 Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 134217728 134217728 Immed
maxtsiz_64bit 1073741824 1073741824 Immed
maxuprc 7372 nproc*9/10 Immed
msgmni 8192 nproc Immed
msgtql 8192 nproc Immed
nflocks 20480 20480 Imm (auto disabled)
ninode 67584 8*nproc+2048
nkthread 14352 nproc*7/4+16 Immed
nproc 8192 8192 Immed
semmni 8192 nproc
semmns 16384 16384
semmnu 8188 nproc-4
shmmax 27487790694 27487790694 Immed
shmmni 1024 1024 Immed
shmseg 360 360 Immed
swchunk 8900 8900
vps_ceiling 64 64 Immed

Glance Version: C.04.55.00

[Rajesh K Rajan]

The results of the 306779.c:

phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 751040 2.9g
cfree mem: 751720 2.9g
total kernel: 4873564 18.6g
kern dyn mem: 2225206 8.5g
kern misc: 2241158 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156218 610.2m
RSS mem: 2538114 9.7g
active RSS mem: 1303087 5.0g
user mem calc: 2556721 9.8g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes

Kernel Classification Breakdowns:
kernel total: 2804933 10.7g
kern static: 17199 67.2m
kern dynamic: 2497466 9.5g
kern Lg Pg: 290198 1.1g
fs metadata: 70 280.0k

[Don Morris_1]

Not, 100%, but add:

print(" bufcache pages:", (uint64_t) pstci.psc_pgcount[PC_BCACHE],
stat.page_size);

around line 172 and re-run.

It looks to me on this limited data (and keep in mind, first this isn't exactly a well cooked pstat program and second, I can't get a breakdown within kernel dynamic) that there's been a lot of inode or direct buffer writes to the disk. There isn't currently much in the buffer cache, but I strongly suspect that that's where a good chunk of the "kernel misc" is -- free pages cached in the arena layer that don't get reported as general dynamic because they're only for buffer cache. That type of activity can also cause a bunch of filesystem metadata, which I suspect is what drove up the kern dynamic. (1.1Gb of it is just large page translation cache stuff -- it will get freed up as things garbage collect or used for new allocations... just a mid-level cache. You aren't under memory pressure, so that reclamation isn't in any hurry).

If PC_BCACHE doesn't show several Gb even with the buffer cache pstat showing only Mb (i.e. the cache has burned through several Gb of buffers but is now only a few Mb of active cache... the free buffers are being held in a cache behind _that_ (object level caching)) then I'll have to punt to getting kmeminfo output, sorry. (Well, I could try to walk you through parsing this stuff in kwdb.. but kmeminfo would be a lot easier).

[Rajesh K Rajan]

The results with the new addition:
phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 1046815 4.0g
cfree mem: 1046390 4.0g
total kernel: 4582793 17.5g
kern dyn mem: 1934979 7.4g
kern misc: 2240614 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156631 611.8m
RSS mem: 2532697 9.7g
active RSS mem: 1204120 4.6g
user mem calc: 2551304 9.7g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes

Kernel Classification Breakdowns:
bufcache pages: 1025611 3.9g
kernel total: 2514354 9.6g
kern static: 17199 67.2m
kern dynamic: 2237658 8.5g
kern Lg Pg: 259427 1013.4m
fs metadata: 70 280.0k

Don, as for the kmeminfo... I do not have the root access to this box.
I administer the databases, and am encountering some serious performance bottlenecks which I can drill down to the OS layer, and in addition when kernel mem is this high, I'm unable to add more memory to the SGA & PGA regions.
My SA is already overloaded, and doesn't have cycles for this, and I want to address this ASAP.

So, a few more questions:
Our Oracle datafile filesystems are all vxfs, and we had enabled the DB to use any means to write to the disks using filesystemio_option=setall.
Since ASYNC doesn't work with vxfs & directio is not enabled, are we being bottlenecked by buffered i/o?
Could our low setting of filecachemax @ 2%of RAM cause my problems?
BTW, databases running on this box are extreme I/O customers, single block and multiblock I/Os.
Let me know if you'd still need the kmeminfo output.

[Rajesh K Rajan]

Gentlemen,

Thanks for your time on this one.

Do you have any update based on the last set of info furnished?

Thanks.