Extremely High System Memory Usage

[Tim Walsh]

We are experiencing extremely high system memory usage in our 11.11 environment. Glance and GPM reports anywhere from 40-75% of our memory is being consumed by the system. The servers are being used primarily as application servers running Apache and Java. When these processes are shut down we see a noticable decrease in user memory but system memory remains high.

Is there any way to determine what is consuming massive amounts of system memory? Are there any know patches which address high system memory usage on 11.11?

[Michael Tully]

There are a number of patches that contain fixes to memory related problems. Start here and do a search for "memory leak"
http://www2.itrc.hp.com/service/patch/search.do

Also, how much buffer cache is set for your system?

[Bill Hassell]

Normally, system memory is consumed by the kernel so a large amount of kernel RAM is usually due to kernel parameters that size tables much too large. An exaple might be nfile and nproc. Did someone set them to hundreds of thousands? Same with ninode. The asiest way to see if common parameters are too large is the kernel is to run Glance and type "t" which shows the system tables. Parameters that are less than 5% utilized on a busy system are far too high. NOTE: If you have VERY small RAM such as 512 megs or less, then high system usage (as a percentage of RAM) is normal but a massive performance hit.

Glance will show the buffer cache as B....B and this is by default far too high (dbc_max_pct=50). Set the percentage to the equivalent of 300-500 megs (and reduce dbc_max_pct if needed down to about 2%).

You can also look in dmesg to see if the initial values from the last bootup are still there. Of interest will be the amount of unlockable memory. This is memory used by the kernel before any applications are started. If this is in the hundreds of megs, then kernel paramters are definitely a suspect. How much RAM do you have?

[Tim Walsh]

Bill and Michael,

Thanks for your responses. The buffer cache seems to be appropriately sized (maybe just a touch large). We have 11GB of physical RAM on the box. The information you have both requested is below. It doesn't seem that the buffer cache is the big consumer of memory. Would you suggest reducing it further or pursuing other parameters such as nfile and nproc? I will also search the patch archive to see if any relative patches jump out at us.

Thanks!
Tim

Parameter Current Dyn Planned
===============================================================================
dbc_max_pct 5 - 5
nfile 18000 - 18000
nproc 7220 - (20+8*MAXUSERS)

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 7220 176 2 2
File Table (nfile) 18010 4255 24 24
Shared Mem Table (shmmni) 200 9 5 5
Message Table (msgmni) 50 2 4 4
Semaphore Table (semmni) 100 26 26 26
File Locks (nflocks) 2000 62 3 3
Pseudo Terminals (npty) 60 0 0 0
Buffer Headers (nbuf) na 124440 na na

Total VM : 6.59gb Sys Mem : 4.08gb User Mem: 4.24gb Phys Mem: 11.0gb
Active VM: 5.90gb Buf Cache: 514.1mb Free Mem: 2.18gb Page 1 of 1

dmesg reports several entries such as:
sysmap_64bit: rmap ovflo, lost [1365099,1365100)

[Con O'Kelly]

Hi Tim

No I wouldn't suggest reducing buffer cache. A 500MB buffer cache on 11i is perfectly normal, especially with 11GB of memory. In any case the system memory reported by glance is independant of buffer cache, so reducing cache will have no impact on system memory.

Your figure of +4GB for system memory does appear extremely high.
The 'nproc' & 'nfile' parameters look OK. From the figures you've given they are probably a little too high but I don't think reducing them will have a significant affect on system memory utilisation.

The errors in dmesg could indicate a memory leak. See doc ID KBRC00000293
http://www1.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchString=KBRC00000293&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&search.x=29&search.y=9

As Michael said it could be a memory leak causing your problem. Can you check if system memory gradually increases over time? What is the system memory utilisation at system boot?
I'd also check the values of the ninode & vx_ninode kernel parameters as extremely large values can utilise alot of system memory.

Cheers
Con