100% Memory Usage

[Jason Moorhead_2]

Hi all,

I have a V-Class with 5GB RAM, with a small 2GB disk swap volume. After a reboot, glance shows me at 100% memory usage, and 90% swap usage, running a Progress database. This is something new within the past few days. I am stumped as to the problem. The only thing I can think of is that maybe the swapmem_on parameter. Any thoughts?

[harry d brown jr]

Do a ls -l /stand/vmunix, is it within the last few days????

swapinfo shows what?

dbc_min&max ?? from kmtune

any errors in dmesg or syslog??


live free or die
harry

[steven Burgess_2]

Hi

Any kernel changes recently ?

Whats your buffer cache set to ?

Regards

Steve

[Patrick Wallek]

The obvious question first: What has changed in the past few days? Have you done a large data load into the DB? Any large reports running? Large queries?

The swapmem_on parameter will not effect the memory usage as glance sees it. Are you REALLY swapping? What does swapinfo show?

The first thing to do is drill down into glance further and see what processes are running that are eating your memory.

[Rich Wright]

Who does your data base maintenance?
Have they added some new instances?

This could be normal. How close were you watching it before the reboot?

Rich

[James R. Ferguson]

Hi Jason:

First, of course, what's changed? Posting the output of 'swapinfo -tam' would be helpful. Do you see memory pressure with 'vmstat -S'?
If you have 'glance', looking at its memory statistics is helpful, too.

Regards!

...JRF...

[Jason Moorhead_2]

dbc_min = 15, max = 20
maxswapchunks = 16384
swapinfo: type=dev (1%) type=memory (82%) type=total (89%)

vmstat shows 0 for 'po'
glance shows 99% Mem Util, 89% Swap Util

[Jason Moorhead_2]

More info - The only change since last week (when everything worked properly) was that HP Apache was installed. The date on vmunix is June 1st, so no problem there. Also, yesturday both Mem Util and Swap Util were at 100% and the system choked itself, and had to be rebooted during production hours. It is quickly approaching 100% again now. It has never done this before.

[Todd Lehr]

Make sure there haven't been any changes to the Progress Database. The buffer setting on the server is a multiple of the block size of the database.

So if you use -B 1000 and the blocksize of the database is 2048, then the memory consumption is 1000*2048

This could have become a problem due to an incorrect Buffer (-B) or if someone has rebuilt/loaded the database with a larger block size.

[James R. Ferguson]

Hi Jason:

One adjustment I'd make is to reduce the buffer cache ceiling ('dbc_max_pct') from 20% (which in your configuration translates to about 1GB RAM) to something like 5-8%. I'd set 'dbc_min_pct' to 2% at the same time. You should see a noticable improvement in memory utilization.

Regards!

...JRF...

[Jason Moorhead_2]

Thanks all. Lowering the dbc_min and max seemed to help quote a bit, as did turning off swapmem_on. We're still running high on utilization, but it looks like we're good now.

Without going with PerfView Alerts, is there a simple way to be alerted when memory utilization gets above a certain point? Does anybody run a cron script that e-mails when a certain level is reached?

[Patrick Wallek]

I think you really want to turn swapmem_on back on!!!!!!

Otherwise with 5GB of RAM and only 2GB of swap, you could run into problems really fast! Since each process that starts has to reserve a bit of swap, once you reach the 2GB of swap reserved, you will NOT be able to start any more processes. If that happens I think you will get MANY complaints!

Another thing you could do if money allows is add more RAM to the machine. 5GB is not a whole lot of RAM anymore, and memory is getting pretty cheap.

[Jason Moorhead_2]

In the last day, I have made some changes that are helping. I've added another GB of RAM (6 total now), and dedicated a whole 9GB drive to swap. Things seem to be okay now, but I'd still like a simple way to monitor mem utilization without buying PerfView Alerts. Anybody have scripts already written?