HPUX 11.31: the "sys mem" is too high, it eats 20% of the physical memory. How can I reduce it?

[Tuan Nguyen_2]

Hi all,

This one is from my note:

Especially on hpux 11.31, the "system memory" is quite high, it occupies up to 20% of your physical memory (using glance, press .m. to check your). Unfortunately the memory DIMM is quite expensive, I can't effort to buy more RAM, how can I reduce the sys memory "without affect the performance".

Here is the way how we reduce this down to about 8% on our database hotel. We reduce the vx_ninode and increase the memory pagesize. Thanks to HP and my colleague.

0) Note the "Sys Mem" value from glance
1) kctune vx_ninode=64000 (quite enough, maybe try to reduce to 32000 for small system)
2) kctune vxfs_ifree_timelag=-1 (one, don.t use time to free inode space, I have enough memory)
3) kctune base_pagesize=16 (default is 4kbytes; each page uses about 200bytes; you get a warning about old java, just ignore it if you don´t using java otherwise you may test it)
4) reboot
read: "Memory Management Subsystem" and "ENW-TBPS-TW.pdf" (Tunable Base Page Size
) for more info. When the pagesize is high, you might risk "the effectiveness of using of memory" may be low. Depend on your application on your server, you can try different value to get the best result.

Check now with glance or kmeminfo (you can find it on the net).


***some facts
before:
[merapi:root]# /nnit/sysadm/tools/kmeminfo -all
tool: kmeminfo 10.13 - libp4 9.495 - libhpux 1.356 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit Montvale on host "merapi"
core: /dev/kmem live kernel!
link: Fri Dec 10 17:03:23 MET 2010
boot: Fri Dec 10 17:09:26 2010
time: Mon Jan 3 08:34:48 2011
nbpg: 4096 bytes (it is the pagesize)

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 16769351 64.0g 100%
Free memory = 12558508 47.9g 75%
User processes = 146030 570.4m 1% details with -user
Detached SHMEM = 51 204.0k 0% details with -shmem
System = 4008088 15.3g 24% (include filecache)
Kernel = 3184981 12.1g 19% kernel text and data (notice 19%)
Dynamic Arenas = 1838976 7.0g 11% details with -arena
vx_inode_kmcach = 270500 1.0g 2%
reg_fixed_arena = 215739 842.7m 1%
spinlock_arena = 215343 841.2m 1%
FCACHE_ARENA = 175144 684.2m 1%
misc region are = 156302 610.6m 1%
Other arenas = 805948 3.1g 5% details with -arena
Super page pool = 167652 654.9m 1% details with -kas
Emergency pool = 33338 130.2m 0% system critical reserve
UAREA's = 12176 47.6m 0%
Overflow pte's = 131181 512.4m 1%
Static Tables = 984022 3.8g 6% details with -static
pfdat = 818815 3.1g 5% (small pagesize -> big pfdat)
vhpt = 131072 512.0m 1%
inode = 14784 57.8m 0%
text = 9155 35.8m 0% vmunix text section
bss = 6747 26.4m 0% vmunix bss section
Other tables = 3446 13.5m 0% details with -static
File Cache = 823107 3.1g 5% details with -ufc


After:
[merapi:root]/ # /nnit/sysadm/tools/kmeminfo
tool: kmeminfo 10.13 - libp4 9.495 - libhpux 1.356 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit Montvale on host "merapi"
core: /dev/kmem live kernel!
link: Fri Dec 10 17:03:23 MET 2010
boot: Wed Jan 5 14:36:45 2011
time: Thu Feb 3 08:18:47 2011
nbpg: 16384 bytes


----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4192335 64.0g 100%
Free memory = 3508939 53.5g 84%
User processes = 145582 2.2g 3% details with -user
Detached SHMEM = 25 400.0k 0% details with -shmem
System = 535914 8.2g 13%
Kernel = 326066 5.0g 8% kernel text and data (sys mem falls to 8% from 19% !!)
Dynamic Arenas = 201473 3.1g 5% details with -arena (no more vx_inode_kmcach and spinlock_arena become leeesser)
vx_buffer_kmcac = 30664 479.1m 1%
FCACHE_ARENA = 18179 284.0m 0%
misc region are = 16717 261.2m 0%
reg_fixed_arena = 15412 240.8m 0%
spinlock_arena = 11549 180.5m 0%
Other arenas = 108952 1.7g 3% details with -arena
Super page pool = 5340 83.4m 0% details with -kas
Emergency pool = 33311 520.5m 1% system critical reserve
UAREA's = 3072 48.0m 0%
Overflow pte's = 8223 128.5m 0%
Static Tables = 67879 1.0g 2% details with -static
pfdat = 51175 799.6m 1%
vhpt = 8192 128.0m 0%
inode = 3696 57.8m 0%
text = 2288 35.8m 0% vmunix text section
bss = 1686 26.3m 0% vmunix bss section
Other tables = 839 13.1m 0% details with -static
File Cache = 209848 3.2g 5% details with -ufc
[merapi:root]/ #

From glance: press "m"
Total VM : 3.0gb Sys Mem : 5.1gb User Mem: 2.2gb Phys Mem : 64.0gb
Active VM: 1.0gb Buf Cache: 3mb Free Mem: 53.6gb FileCache: 3.1gb
MemFS Blk Cnt: 0 MemFS Swp Cnt: 0

"Sys Mem / Phys Mem" = 5.1/64 =.07968750000000000000 = 8% ; before the "sys mem" was 12.1gb (see the "kernel" at the kmeminfo above).

[merapi:root]/ # kctune|grep filecache
filecache_max 3392417792 5% Imm (auto disabled)
filecache_min 1356967116 2% Imm (auto disabled)

[Emil Velez]

filecache_min=1%
filecache_max=2%

You could go through the /etc/rc.config.d/* and turn off any daemons you are not using like

sendmail
ssh
nfs_server
nfs_client
vxvm (if you only use LVM)
ntp

You would be somewhat suprised at all the stuff that is started that you might not need.

What application are you using on that system ? you mention database ? Which one. THere are alot of tuning that you can do with oracle and other databases like the size of the SGA and i/o blocksize.

What processes are using the most memory. Top should show you that. What are they doing and are they necessary.

In the example above you have 47 GB free ? Are you low on memory after the application starts ?

[Rita C Workman]

You reduced the vx_ninode (so glad to see someone doing that!)

ninode is another one that creates a large table in memory based on physical memory size, and it never requires as much as it takes....so....
What is your ninode set at?....You likely can reduce that as well.
Run sar -v 1 10 to get a better view of what it is really using and then tune accordingly.

Regards,
Rita

[Tuan Nguyen_2]

Hi all,

To Emil, It is not really true that I wrote "can't effort to buy more ram" (we just buy more RAM). But one DBA observed "it can´t be true that the OS takes 20% of the total RAM". This case is not to "tune" the server but to find out "why".

>>filecache_min=1%
>>filecache_max=2%
We run cio and (mincache=direct,convosync=direct) on our oracle mountpoints, but still we need having cache enough for documentum/oracle binary/sap and other OS relate processes. This value is interchangeable depending on if this is a database hotel (20 ->80 oracle databases) or an "application/sap" server (with 2, 3 oracle bases). For database hotel, yes we use 1-2%. The values I listed yesterday can be lower.

>>What processes are using the most memory. Top should show you that. What are they doing and are they necessary.
Again it is not the issue. To disable vxvm is a good idea.

>>In the example above you have 47 GB free ? Are you low on memory after the application starts ?
No, this one is just among our other new 11.31 server (with update 7), preparing for a new database hotel.

To Rita:
>>You reduced the vx_ninode (so glad to see someone doing that!)
thanks to HP for that, they suggest ninode as well, but I did´t do it because I think the outcome is not as big (I am going for a GB only) as vx_ninode and memory pagesize as we see with "kmeminfo -all".

Again it is not a question about "tuning" but to understand the "20%". For performance, I use glance/korn script (which takes the samples for a whole week/weekend), kcusage, hpjmeter, vmstat, ps. Yes "sar -v" is good, but I never remember to use it.

..I forgot to write, the (mincache=direct,convosync=direct) for fstab (only for oracle data mountpoints) can reduce your FS cache as well.

Regards,
Tuan

[Rita C Workman]

Well.....let's think about this. A parm like ninode allocates resources based on what the value is that it is set at. If it is really high, the more it grabs. And it doesn't need it, but it is going to grab it and then the system must manage it. Thus taking resources for ninode and resources to manage it.
Run sar -v 1 10 and look at ninode.

I hear what your saying, that your not looking at tuning. But maybe it might be said...how much resources it takes, is how much you let it take.


Rita