topic Re: Sys Mem consuming > 1/3 of Physical RAM in Operating System - HP-UX

Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Thu, 12 Jun 2008 19:55:34 GMT

My Oracle DB Server is "ia64 hp server rx8640", Release: HP-UX B.11.31
It has the following config:
8 Intel(R) Itanium 2 9000 series processors (1.6 GHz, 24 MB)
Firmware info:
Firmware revision: 7.048
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 2.05
Memory: 32568 MB (31.8 GB)

This runs 2 Multi Terra-Byte Oracle 10.2.0.3 databases whose combined SGA takes 16GB and combined PGA takes 2.5 GB.

There is excessive paging observed on this box. One another observation is that GLANCE shows us Sys Mem at 9.5GB or round-abouts consistently.
Isn't this a very excessive value for the kernel memory?
Is this essential and can some of that memory be freed up?

FYI : We have enabled the DB to use ASYNCIO by granting the MLOCK priv to the DBA group, and use FILESYSTEM_IO_OPTIONS=setall, and ASYNC_DISK_IO=TRUE.

Thanks.

Re: Sys Mem consuming > 1/3 of Physical RAM

Tim Nelson — Thu, 12 Jun 2008 20:56:44 GMT

I would agree that 9.5GB of kernel space is rediculos, unless there is some kernel param set really really high..
kmeminfo is a great tool to help isolate. ( found on this forum)

As far as swapping/pageouts are concerned, there is still about 4GB left of free ( or at least unaccounted for ).

What does glance say about free mem ?
What does swapinfo say about swap usage ?
What does glance say about deactivations ?

If free mem is truely zero then you need to go searching for who is using up the rest, how to reduce the system usage, and / or make sure you have enough swap enabled so all your RAM can be used.

Start posting some stats and maybe we can guide you.

T.

Re: Sys Mem consuming > 1/3 of Physical RAM

Bill Hassell — Fri, 13 Jun 2008 01:41:42 GMT

There are several kernel parameters that can make the kernel space very large. 11.31 is a lot better at allocating table space dynamically but take a look at all the parameters that were changed from default. Make sure the fixed parameters are not excessively large (ie, only a small portion is actually used)

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Fri, 13 Jun 2008 03:17:33 GMT

Results from Glance:
Total VM : 13.4gb Sys Mem : 9.7gb User Mem: 18.5gb Phys Mem : 31.8gb
Active VM: 7.3gb Buf Cache: 1mb Free Mem: 3.0gb FileCache: 616mb

Results from swapinfo -t:
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4191900 0% 0 - 1 /dev/vg00/lvol2
dev 70352896 199064 70146536 0% 0 - 0 /dev/vg00/lvol8
dev 70352896 198848 70146752 0% 0 - 0 /dev/vg00/lvol9
reserve - 20958284 -20958284
memory 31721764 25726780 5994984 81%
total 176621860 47082976 129521888 27% - 0 -

Using a tool from the forum (memdetail):
Memory Stat total used avail %used
physical 32568.9 29589.3 2979.6 91%
active virtual 13905.5 7944.4 5961.0 57%
active real 10770.8 5431.1 5339.7 50%
memory swap 30978.3 25126.2 5852.1 81%
device swap 141487.4 20472.7 121014.7 14%

As for the filecache settings:
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
Since this is a DB server and we're using ASYNCIO we'd set the FILECACHE to the minimum.

The database files are on vxfs filesystems something like below from the fstab:
/u01/dat10 vxfs largefiles,delaylog,nodatainlog 0 2

Let me know if I can provide additional info.

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Sat, 14 Jun 2008 21:47:09 GMT

Latest results from MEMDETAIL:
Memory Stat total used avail %used
physical 32568.9 29164.6 3404.4 90%
active virtual 12221.1 3410.9 8810.3 28%
active real 10152.6 1998.2 8154.4 20%
memory swap 30978.3 25221.1 5757.2 81%
device swap 141487.4 19456.1 122031.3 14%

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Thu, 19 Jun 2008 17:59:06 GMT

Folks,

Any help is appreciated.....

Re: Sys Mem consuming > 1/3 of Physical RAM

Tim Nelson — Thu, 19 Jun 2008 18:57:29 GMT

Can u post the output of kctune -S

please.

Also, on 11.31 glance should report buff cache at 0 (zero) as it is obsoleted. What version of glance are u running ?

kmeminfo ouputs would also help.

Re: Sys Mem consuming > 1/3 of Physical RAM

Don Morris_1 — Thu, 19 Jun 2008 20:08:57 GMT

I think Glance is reporting the Filesystem Metadata area as Buffer cache (because that's what the pstat field labeled buffer cache now is).

As a first pass you can run the attached, but if it just basically boils down to a bunch of kernel dynamic, there's really no way to break things down without using a Support tool like kmeminfo. (Have you tried talking to support? Yes, I really hate recommending kmeminfo be run willy-nilly... but probably about the only way to break this down.)

On a side note, are you really paging (with 3Gb of RAM free, you shouldn't be) -- or is vhand having to sync dirty pages out of the file cache and that's what you're seeing as po in vmstat? With a 1 to 2% file cache that's showing as full size -- it wouldn't surprise me if you can easily get into cache thrashing. If you've got 3Gb of RAM laying around, it is worth considering giving the file cache some breathing room there.

Last but not least -- what's your VM patch level so we have context? (I'd recommend the 11.31.0803 level of PHKL_37452 if you aren't there, lots of vhand/filecache interaction adjustments through that patch stream).

Re: Sys Mem consuming > 1/3 of Physical RAM

Don Morris_1 — Thu, 19 Jun 2008 20:09:34 GMT

Re: Sys Mem consuming > 1/3 of Physical RAM

Don Morris_1 — Thu, 19 Jun 2008 20:11:41 GMT

Excuse the double post there -- realized I'd forgotten to append the .c file and hit stop -- but apparently not fast enough.

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Thu, 19 Jun 2008 21:14:35 GMT

KCTUNE -S :
Tunable Value Expression Changes
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
ksi_alloc_max 65536 nproc*8 Immed
max_thread_proc 1200 1200 Immed
maxfiles 20480 20480
maxfiles_lim 20480 20480 Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 134217728 134217728 Immed
maxtsiz_64bit 1073741824 1073741824 Immed
maxuprc 7372 nproc*9/10 Immed
msgmni 8192 nproc Immed
msgtql 8192 nproc Immed
nflocks 20480 20480 Imm (auto disabled)
ninode 67584 8*nproc+2048
nkthread 14352 nproc*7/4+16 Immed
nproc 8192 8192 Immed
semmni 8192 nproc
semmns 16384 16384
semmnu 8188 nproc-4
shmmax 27487790694 27487790694 Immed
shmmni 1024 1024 Immed
shmseg 360 360 Immed
swchunk 8900 8900
vps_ceiling 64 64 Immed

Glance Version: C.04.55.00

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Thu, 19 Jun 2008 21:19:37 GMT

The results of the 306779.c:

phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 751040 2.9g
cfree mem: 751720 2.9g
total kernel: 4873564 18.6g
kern dyn mem: 2225206 8.5g
kern misc: 2241158 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156218 610.2m
RSS mem: 2538114 9.7g
active RSS mem: 1303087 5.0g
user mem calc: 2556721 9.8g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes

Kernel Classification Breakdowns:
kernel total: 2804933 10.7g
kern static: 17199 67.2m
kern dynamic: 2497466 9.5g
kern Lg Pg: 290198 1.1g
fs metadata: 70 280.0k

Re: Sys Mem consuming > 1/3 of Physical RAM

Don Morris_1 — Fri, 20 Jun 2008 01:05:26 GMT

Not, 100%, but add:

print(" bufcache pages:", (uint64_t) pstci.psc_pgcount[PC_BCACHE],
stat.page_size);

around line 172 and re-run.

It looks to me on this limited data (and keep in mind, first this isn't exactly a well cooked pstat program and second, I can't get a breakdown within kernel dynamic) that there's been a lot of inode or direct buffer writes to the disk. There isn't currently much in the buffer cache, but I strongly suspect that that's where a good chunk of the "kernel misc" is -- free pages cached in the arena layer that don't get reported as general dynamic because they're only for buffer cache. That type of activity can also cause a bunch of filesystem metadata, which I suspect is what drove up the kern dynamic. (1.1Gb of it is just large page translation cache stuff -- it will get freed up as things garbage collect or used for new allocations... just a mid-level cache. You aren't under memory pressure, so that reclamation isn't in any hurry).

If PC_BCACHE doesn't show several Gb even with the buffer cache pstat showing only Mb (i.e. the cache has burned through several Gb of buffers but is now only a few Mb of active cache... the free buffers are being held in a cache behind _that_ (object level caching)) then I'll have to punt to getting kmeminfo output, sorry. (Well, I could try to walk you through parsing this stuff in kwdb.. but kmeminfo would be a lot easier).

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Fri, 20 Jun 2008 19:11:24 GMT

The results with the new addition:
phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 1046815 4.0g
cfree mem: 1046390 4.0g
total kernel: 4582793 17.5g
kern dyn mem: 1934979 7.4g
kern misc: 2240614 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156631 611.8m
RSS mem: 2532697 9.7g
active RSS mem: 1204120 4.6g
user mem calc: 2551304 9.7g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes

Kernel Classification Breakdowns:
bufcache pages: 1025611 3.9g
kernel total: 2514354 9.6g
kern static: 17199 67.2m
kern dynamic: 2237658 8.5g
kern Lg Pg: 259427 1013.4m
fs metadata: 70 280.0k

Don, as for the kmeminfo... I do not have the root access to this box.
I administer the databases, and am encountering some serious performance bottlenecks which I can drill down to the OS layer, and in addition when kernel mem is this high, I'm unable to add more memory to the SGA & PGA regions.
My SA is already overloaded, and doesn't have cycles for this, and I want to address this ASAP.

So, a few more questions:
Our Oracle datafile filesystems are all vxfs, and we had enabled the DB to use any means to write to the disks using filesystemio_option=setall.
Since ASYNC doesn't work with vxfs & directio is not enabled, are we being bottlenecked by buffered i/o?
Could our low setting of filecachemax @ 2%of RAM cause my problems?
BTW, databases running on this box are extreme I/O customers, single block and multiblock I/Os.
Let me know if you'd still need the kmeminfo output.

Re: Sys Mem consuming > 1/3 of Physical RAM

Rajesh K Rajan — Tue, 24 Jun 2008 22:25:25 GMT

Gentlemen,

Thanks for your time on this one.

Do you have any update based on the last set of info furnished?

Thanks.