Sys Mem consuming > 1/3 of Physical RAM

[Rajesh K Rajan]

My Oracle DB Server is "ia64 hp server rx8640", Release: HP-UX B.11.31
It has the following config:
8 Intel(R) Itanium 2 9000 series processors (1.6 GHz, 24 MB)
Firmware info:
Firmware revision: 7.048
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 2.05
Memory: 32568 MB (31.8 GB)

This runs 2 Multi Terra-Byte Oracle 10.2.0.3 databases whose combined SGA takes 16GB and combined PGA takes 2.5 GB.

There is excessive paging observed on this box. One another observation is that GLANCE shows us Sys Mem at 9.5GB or round-abouts consistently.
Isn't this a very excessive value for the kernel memory?
Is this essential and can some of that memory be freed up?

FYI : We have enabled the DB to use ASYNCIO by granting the MLOCK priv to the DBA group, and use FILESYSTEM_IO_OPTIONS=setall, and ASYNC_DISK_IO=TRUE.

Thanks.

[Tim Nelson]

I would agree that 9.5GB of kernel space is rediculos, unless there is some kernel param set really really high..
kmeminfo is a great tool to help isolate. ( found on this forum)

As far as swapping/pageouts are concerned, there is still about 4GB left of free ( or at least unaccounted for ).

What does glance say about free mem ?
What does swapinfo say about swap usage ?
What does glance say about deactivations ?

If free mem is truely zero then you need to go searching for who is using up the rest, how to reduce the system usage, and / or make sure you have enough swap enabled so all your RAM can be used.

Start posting some stats and maybe we can guide you.

T.

[Bill Hassell]

There are several kernel parameters that can make the kernel space very large. 11.31 is a lot better at allocating table space dynamically but take a look at all the parameters that were changed from default. Make sure the fixed parameters are not excessively large (ie, only a small portion is actually used)

[Rajesh K Rajan]

Results from Glance:
Total VM : 13.4gb Sys Mem : 9.7gb User Mem: 18.5gb Phys Mem : 31.8gb
Active VM: 7.3gb Buf Cache: 1mb Free Mem: 3.0gb FileCache: 616mb

Results from swapinfo -t:
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4191900 0% 0 - 1 /dev/vg00/lvol2
dev 70352896 199064 70146536 0% 0 - 0 /dev/vg00/lvol8
dev 70352896 198848 70146752 0% 0 - 0 /dev/vg00/lvol9
reserve - 20958284 -20958284
memory 31721764 25726780 5994984 81%
total 176621860 47082976 129521888 27% - 0 -

Using a tool from the forum (memdetail):
Memory Stat total used avail %used
physical 32568.9 29589.3 2979.6 91%
active virtual 13905.5 7944.4 5961.0 57%
active real 10770.8 5431.1 5339.7 50%
memory swap 30978.3 25126.2 5852.1 81%
device swap 141487.4 20472.7 121014.7 14%

As for the filecache settings:
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
Since this is a DB server and we're using ASYNCIO we'd set the FILECACHE to the minimum.

The database files are on vxfs filesystems something like below from the fstab:
/u01/dat10 vxfs largefiles,delaylog,nodatainlog 0 2

Let me know if I can provide additional info.

[Rajesh K Rajan]

Latest results from MEMDETAIL:
Memory Stat total used avail %used
physical 32568.9 29164.6 3404.4 90%
active virtual 12221.1 3410.9 8810.3 28%
active real 10152.6 1998.2 8154.4 20%
memory swap 30978.3 25221.1 5757.2 81%
device swap 141487.4 19456.1 122031.3 14%

[Rajesh K Rajan]

Folks,

Any help is appreciated.....

[Tim Nelson]

Can u post the output of kctune -S

please.

Also, on 11.31 glance should report buff cache at 0 (zero) as it is obsoleted. What version of glance are u running ?

kmeminfo ouputs would also help.

[Don Morris_1]

I think Glance is reporting the Filesystem Metadata area as Buffer cache (because that's what the pstat field labeled buffer cache now is).

As a first pass you can run the attached, but if it just basically boils down to a bunch of kernel dynamic, there's really no way to break things down without using a Support tool like kmeminfo. (Have you tried talking to support? Yes, I really hate recommending kmeminfo be run willy-nilly... but probably about the only way to break this down.)

On a side note, are you really paging (with 3Gb of RAM free, you shouldn't be) -- or is vhand having to sync dirty pages out of the file cache and that's what you're seeing as po in vmstat? With a 1 to 2% file cache that's showing as full size -- it wouldn't surprise me if you can easily get into cache thrashing. If you've got 3Gb of RAM laying around, it is worth considering giving the file cache some breathing room there.

Last but not least -- what's your VM patch level so we have context? (I'd recommend the 11.31.0803 level of PHKL_37452 if you aren't there, lots of vhand/filecache interaction adjustments through that patch stream).

[Don Morris_1]

I think Glance is reporting the Filesystem Metadata area as Buffer cache (because that's what the pstat field labeled buffer cache now is).

As a first pass you can run the attached, but if it just basically boils down to a bunch of kernel dynamic, there's really no way to break things down without using a Support tool like kmeminfo. (Have you tried talking to support? Yes, I really hate recommending kmeminfo be run willy-nilly... but probably about the only way to break this down.)

On a side note, are you really paging (with 3Gb of RAM free, you shouldn't be) -- or is vhand having to sync dirty pages out of the file cache and that's what you're seeing as po in vmstat? With a 1 to 2% file cache that's showing as full size -- it wouldn't surprise me if you can easily get into cache thrashing. If you've got 3Gb of RAM laying around, it is worth considering giving the file cache some breathing room there.

Last but not least -- what's your VM patch level so we have context? (I'd recommend the 11.31.0803 level of PHKL_37452 if you aren't there, lots of vhand/filecache interaction adjustments through that patch stream).

[Don Morris_1]

Excuse the double post there -- realized I'd forgotten to append the .c file and hit stop -- but apparently not fast enough.