Interesting issue with the buffer cache.

[Steven E. Protter]

I got a guestion on buffer cache, did some digging and ended up with a bigger question.

# Yeah we have a sense of humor with system names. This box is mine.
[root@baghdad kernel]# free -m
total used free shared buffers cached
Mem: 122 120 2 0 32 42
-/+ buffers/cache: 44 77
Swap: 255 19 236

I've noticed an awful lot of our active boxes have the cache set around 25%. I wondered why this was and if I could change it to make my system more efficient.

RH 4 update 4 Centos 4 update 4.

A google search came up with some ways of controlling buffer cache size in legacy versions of RedHat.

http://www.faqs.org/docs/securing/chap6sec69.html
6.2 or 7.1 had a kernel parameter called buffermem.min_percent and buffermem.max_percent or buffermem with three parameters.

HP-UX has dbc_max_pct and dbc_min_pct to permit dynamic control of the size of the buffer cache in the OS.

Can't seem to find a mechanism for setting this in RedHat4 update 4/Centos 4 update 4.

I'd really like to know the answer to these questions:

1) Can buffer cache be controlled?
2) How?
3) If not who made this decision and why?

Thanks,

SEP

[Stuart Browne]

I've lost count of the number of times I've looked into this topic.

I'm yet to get a working answer.

The only things I've been able to find are messing with the age of the buffer/cache content, and even then, that wasn't too useful.

The 2.6 series of kernel (as far as I can tell) doesn't give you control over them.

What it does give you is multiple IO schedulers: http://www.redhat.com/magazine/008jun05/features/schedulers/

From the other side, memory management side, I got used to SCO OSR's ability to limit buffers as well.

So, in short:

1) I don't beleive so.
2) *buzz*
3) Linus/Kernel-dev team.

Interesting reading is in 'systcl/vm.txt' in the kernel documentation.

[Ivan Ferreira · Accepted solution]

1) Yes
2) Tune bdflush
http://www.redhat.com/magazine/001nov04/features/vm/
3) See 1)

[Steven E. Protter]

I'm looking for a system to test this on.

I'd definitely reward an post that showed how one would go about limiting the buffer cache to 10% of RAM if one wished to do that. Likely a point opportunity while I look for a system to test this on in my lab.

SEP

[Ralph Grothe]

Hi,

from what I have read Linux (current kernels) assign almost all unused memory as buffer cache.
But unlike other memory management strategies (e.g. hpux's) these pages will be released quite easily if the apps or system require more memory.
I cannot recall where I have read it.
But I suspect a book about about the 2.6 kernel that I may have at home.
Maybe I could quote from there on Monday if I find the stanzas?

[Heironimus]

I think a patch was submitted not long ago to let 2.6 kernels limit the page cache size to a percentage, but even if that makes it in to 2.6.20 you probably will never see it in CentOS 4 or RHEL 4 (unless they backport the patch in one of the quarterly updates).

But most of the time you don't need it. Growing and shrinking the page cache on the fly should be pretty cheap in the 2.6 kernel, so there are only a few situations where you would see a tangible gain. Sitting at ~98% total memory utilization is pretty common for Linux systems and does not cause the kind of pressure that you would see on a 9000 in that same situation.

You probably want to look at tuning how/when write buffers get flushed to disk. The data that's written won't be removed from memory until a process needs it - the page just gets marked as not being dirty anymore. If you have too many dirty buffers a sudden big request for memory could force a lot of buffers to be flushed at once, which would be a big hit.

[Steven E. Protter]

Ralph,

Feel free to post anything you want on Monday. It will be rewarded accordingly.

I've never had a problem with the buffer cache on Linux. Its certainly not causing me any performance issues at this point. When there is application memory pressure on the effected machines they respond accordingly.

Some people think this kernel strategy is wasteful.

I'm from the HP-UX school where you limited buffer cache changes because while they were being executed the system could suffer from serious performance problems.

SEP

[Ragu_3]

SEP, "vmstat" is your friend. The "si" and "so" values indicate buffer states.

[Vitaly Karasik_1]

>http://www.redhat.com/magazine/001nov04/features/vm/

as far as I see, bdflush kernel thread and /proc/sys/vm/bdflush don't exist in 2.6 kernel (RHEL4).

[Steven E. Protter]

They have made life a little hard haven't they Vitaly.

Toda Raba lachem

Thank you all.

Continued input welcome.

SEP

[Steven E. Protter]

Got an inquiry and have to admit its a wild one.

Someone I know who is clearly unfamiliar with me had a system.

12 Gb of RAM. Buffer cache was at exactly 25% of physical memory. There was no memory pressure at all and the system was nearly idle because it was not production.

I was consulted on the job, hence this thread.

The conclusion I sent back was that Linus had taken control of the buffer cache and if he wished we could manipulate it.

But I also asked why? Was there a performance problem or are we just annoyed at memory use.

I believe from experience and your input that if an application needs memory allocated to the buffer cache under the 2.6 kernel it will get it. My tests show it will get the memory immediately with no negative impact on performance as opposed to HP-UX which under the same circumstances takes a nasty perofmrance hit(hpux 11.11).

The reaction was bizaare at the other end of this thread.

They decided to disable swapping altogether, because the system was not under memory pressure. The buffer cache ballooned to a huge figure.

I warned them that every process reserves swap when it opens in case it should need to be swapped in the future. Disabling swap was dangerous and could lead to the system suddenly ceaseing operation. I thought this was reasonable since the installation program warns you about this if you try and set up a system without swap.

I'm asking for experiences and links.

My last suggestion was to run a load test on the system and see if it crashes.

Regards,

Shmuel

[Ivan Ferreira]

Other interesting readings:

Very good!
http://people.redhat.com/nhorman/papers/rhel4_vm.pdf

nice
http://www.redhat.com/f/summitfiles/presentation/May31/Core%20OS/Woodman,Shakshober_Performance%20Analys.pdf