buffer cache goes up, swap usage, then no free mem

[Chris Watson]

I had buffer cache set to 10-50%. During my overnight backup, the buffercache grows to 50%. When users come online at 8am, buffercache is reducing very slowly, if at all, and this causes swapping out to disk.

The swapping caused 20 second hangs every ten minutes.

I have reduced buffercache to 20-20% and now I do not do any swapping. No more hangs.

However, I now have 20% free memory at peaks times. It seems that it could be utilised as buffercache, but as it doesn't reduce quickly, AND causes swapping I am lost.

How do I get buffer cache dynamic reduction to occur BEFORE swapping to disk? Turn off swap?

[Steven E. Protter]

Do not turn off swap. Device swap is best.

You have stopped the hangs, thats a great start.

Now its time to use the script I'm attaching to collect some data and plan your next move.

SEP

[Stefan Farrelly]

I dont believe there is any way to force buffer cache to reduce before memory pressure causes it to (and thus swapping has already started). You can write a simple C program which simply locks itself into memory using X MB, and this will cause buffer cache to reduce, but without user processes using this free memory quickly buffer cache will simply increase again leaving you with the same problem.

Do what most other people do - dont allow dynamic buffer cache, set it to a limit of X% which equates to Z MB. Thats what we do. If buffer cache is greate than around 400MB then it slows down performance a little anyway so we limit ours to 400MB.

[A. Clay Stephenson]

Since you didn't bother to identify your OS or the amount of physical memory that you have, it's difficult to be precise. HP-UX 11.11 is much better than previous versions at free space used by the buffercache. In almost everycase, 50% is much too much and values above 20% seldom realize any gains in performance. Your first task should be to check for any patches that should be applied dealing with buffer cache and/or filesystems.

Does swapinfo show any actual dev/filesystem swapping or it it all "memory/pseudoswap" which isn't swapping at all.

If you are going to use a 20-20% setting which may not be bad or any other dynamic buffer cache range where the delta is small, you might as well eliminate that bit of kernel overhead and used a fixed buffer cache by setting bufpages to a non-zero value. That's my preference.

Assuming that you have lots of memory here are my rules of thumb:

10.20 -- 200-400MB
11.0 -- 400-800MB
11.11 -- 800-1200MB

Buffer cache above these ranges tend to yield diminishing results. Bear in mind that there are cases where the time required to search large buffer cache can exceed the time to fetch the data from disk --- especially with modern arrays that already have lots of cache.

[Chris Watson]

Stefan,

I am still thinking of getting rid of fs swap. Do you think that as free memory approaches 0% that buffer cache would now reduce quickly?

ACS,

It was swapping to disk. I was a little suprised at how much it affected performance when the swapping was actually taking place. The system was hanging user processes for twenty seconds. I was rather more familiar with Windows workstation swapping; slowing down the system but not grinding to a halt.

I am running 11.11 with 2G memory. Therefore before I had 1G buffer in use (which went down very slowly indeed), 400 MB active swap, and 0 free mem. I will take up your suggestion on bufpages, sticking to around 400MB for now. Although still, it do now now have 200-400MB free memory at all times. If buffer was dynamic enough in reducing I could set it higher.

Re my comment to Stefan. Would getting rid of swap space make a dynamic buffer reduce quickly enough to stop out of memory condition?

Thanks all.

[keith persons]

Chris

As for releasing cache, there's no fixed formula or rule of thumb in hp-ux altho I believe the later versions are a bit better. The speed with which the cache is released is a function of several variables. The most agonizing scenario is memory is allocated to cache due to an overnight backup by root when there's little competition for resources. Under these circumstances, the cache is released at a painfully slow rate.

You can try to influence cache management some by altering mainly the lotsfree parm in the master config file. You are in essences trying to unfluence scheduling for the vhand process. Lotsfree is the first low-water mark for cache and is usually around 100mb. However, realize the installation of any kind of update on the system that impacts the kernel may revert lotsfree to the default value. And, while it should work, there are no guarantees and it will require a great deal of trial and error to see if values other than the default provide any improvement.

I also recommend removing any file system swap and use device/psuedo-swap at least the size of your memory. Also, make sure there is only one swap area per disk/lun and 2 equal size swap devices will typically perform better than one large swap device.

Keith