Unreasonable kernel caching

[david_69]

Hey guys,
I got this farm of computers ( redhat 7.3 linux 2.4.18 compaq DL360). They have apache and tomcat installed.
On some days the kernel cache grows to a certain point ( bringing total mem usage upto about 95%) , then it just overs around 9% for the day until the logrotate at 4am and the cache is "cleared". Somedays however the cache NEVER levels out, memory usage goes to 98%-99% and network services stop responding. Stopping and starting network services does not "free up" the memory.
Swaping is not happening.
/proc/sys/fs/file-max is 115000.
Web usage is not particularly varied; +- 10% in any given day with no clear correlation between traffic patterns and "crashes".
Over commit memory is not set.
So I have two questions:
Why can't I trust the kernel to mamange memory effectivly?
Why do I see this sometimes and not other times?
I have a graph of a good day:
http://www.quadratic.net/~david/gnuplot/#Sar_Analysis
On a bad day the dark blue line just goes up and up and up.
I have heard: you need more memory and I don't belive it.
It's not my processes that are using up memory. The machiens has 2 GB RAM now.
Lsof shows that larges open files as the web server log files at 400-1000GB in size.
My best guess as to a fix is to get logrotation to run twice a day rather than once a day.
Also I have played around with hdparm -f /dev/cciss/c0d0pX
( yes I know that hdparm is for ide drives but I can flush the cache on /dev/sda1 with it ), unfortunately at this time hdparm-5.2-1 "doesn't support that device" (I am working on a patch now).

david

[Steven E. Protter]

If the servers aren't doing anything in particular, they are going to allocate memory based on what they've done in the past.

It is quite common for large chunks of physical memory to be allocated for database buffers and such. If there are other needs for this memory, it will be drawn down and used in other ways.

You should have a good log rotation policy where old logs cut cut off and copied to tape and whatnot and new logs get created. This will minimize the amount of resources being used for this purposes.

Once a week we do log switching on our Linux and HP-UX servers for reasons of disk space control.

SEP

[david_69]

Thanks for the input.

I am rotating logs daily already. I have masses of disk space ( 26GB Free)

[U.SivaKumar_2]

Hi,

Set the limit of buffercache to grow to only 50% of the memory.

#echo "2 10 50" > /proc/sys/vm/buffermem

If your problem does not repeat make this entries permanent by editing /etc/sysctl.conf.

regards,

U.SivaKumar