system and buffer memory increase

[Ian Lochray]

We produce a large application that comprises a number of executables and shared libraries. Some of our "core" code is built into small archive libraries (about 1/100th of the entire product size) and these are linked into the executables. The idea is that the "core" code is always available within the executables.
I have just tried relinking these archive libraries as shared libraries and relinking the executables. I then ran a benchmark for the before and after code and measured memory usage using glance. I saw that the user memory decreased by about 5% (50Mb) but system memory increased by 100% (220Mb) and buffer cache increased by 250% (180Mb). Total memory usage for the archive version was 80% (of 1500Mb) and 100% for the entirely shared version.
Can someone explain the large increase in system memory and buffer cache.

[Bill Hassell]

The shared libraries are designed for multiple copies of the same program. Run 10 copies of the same program using archive libraries and you'll see a very large amount of additional RAM now being used. Depending on the structure of the shared library, some unused routines may be loaded into RAM when a specific application does not need them and this will take system memory as the libraries are systemn resources.

The buffer cache is not directly by shared libraries except as the file are read into RAM, they will also be stored in the buffer cache in anticipation of future reads. Since an archived executable is self-contained, the library code is read directly from the executable.

The concept of archived libraries is to be self-contained (a good thing in single user mode when /usr is not mounted) but wastes RAM when dozens of copies of the same program are run at the same time. If the app is never run as multiple instances, then there is little difference except as you've seen.

[Michael Steele_2]

Very nice document under /usr/share/docs/mem_mgt.txt.

[Ian Lochray]

Bill,
I appreciate what you say about the design of shared libraries. Our application does have multiple copies of the same program running. This explained to me why the user memory usage decreased when I ran the shared library only version. I am happy as to why the buffer cache usage rose but I still do not know why the system memory usage increased by so much.

[Jeff Schussele]

Hi Ian,

Several things to remember about dynamic buffer cache:

1) The minimum (dbc_min_pct) is of course a static amount of memory, but it is also part of the kernel memory. Therefore great care should be taken to not set it too large as it will increase kernel size as well as lockable memory size.

2) The maximum (dbc_max_pct) is a "floating" value & will be pushed towards it's limit by heavy disk I/O - especially large reads & writes. But it will ONLY go down when RAM usage pressures it down. So again great care should be taken to poperly size it as well.

3) There is an overhead cost for using buffer cache & it's CPU usage. The CPU has to monitor & manipulate the cache, so there will be times when it's better to not use the buffer - mainly when reads & writes are in LARGE chunks.

My 2 cents,
Jeff