topic Re: 100% Memory Usage in Operating System - HP-UX

100% Memory Usage

Jason Moorhead_2 — Tue, 27 Aug 2002 17:37:59 GMT

Hi all,

I have a V-Class with 5GB RAM, with a small 2GB disk swap volume. After a reboot, glance shows me at 100% memory usage, and 90% swap usage, running a Progress database. This is something new within the past few days. I am stumped as to the problem. The only thing I can think of is that maybe the swapmem_on parameter. Any thoughts?

Re: 100% Memory Usage

harry d brown jr — Tue, 27 Aug 2002 17:42:35 GMT

Do a ls -l /stand/vmunix, is it within the last few days????

swapinfo shows what?

dbc_min&max ?? from kmtune

any errors in dmesg or syslog??

live free or die
harry

Re: 100% Memory Usage

steven Burgess_2 — Tue, 27 Aug 2002 17:44:24 GMT

Hi

Any kernel changes recently ?

Whats your buffer cache set to ?

Regards

Steve

Re: 100% Memory Usage

Patrick Wallek — Tue, 27 Aug 2002 17:45:07 GMT

The obvious question first: What has changed in the past few days? Have you done a large data load into the DB? Any large reports running? Large queries?

The swapmem_on parameter will not effect the memory usage as glance sees it. Are you REALLY swapping? What does swapinfo show?

The first thing to do is drill down into glance further and see what processes are running that are eating your memory.

Re: 100% Memory Usage

Rich Wright — Tue, 27 Aug 2002 17:45:17 GMT

Who does your data base maintenance?
Have they added some new instances?

This could be normal. How close were you watching it before the reboot?

Rich

Re: 100% Memory Usage

James R. Ferguson — Tue, 27 Aug 2002 17:54:22 GMT

Hi Jason:

First, of course, what's changed? Posting the output of 'swapinfo -tam' would be helpful. Do you see memory pressure with 'vmstat -S'?
If you have 'glance', looking at its memory statistics is helpful, too.

Regards!

...JRF...

Re: 100% Memory Usage

Jason Moorhead_2 — Tue, 27 Aug 2002 17:56:29 GMT

dbc_min = 15, max = 20
maxswapchunks = 16384
swapinfo: type=dev (1%) type=memory (82%) type=total (89%)

vmstat shows 0 for 'po'
glance shows 99% Mem Util, 89% Swap Util

Re: 100% Memory Usage

Jason Moorhead_2 — Tue, 27 Aug 2002 18:03:24 GMT

More info - The only change since last week (when everything worked properly) was that HP Apache was installed. The date on vmunix is June 1st, so no problem there. Also, yesturday both Mem Util and Swap Util were at 100% and the system choked itself, and had to be rebooted during production hours. It is quickly approaching 100% again now. It has never done this before.

Re: 100% Memory Usage

Todd Lehr — Tue, 27 Aug 2002 18:03:30 GMT

Make sure there haven't been any changes to the Progress Database. The buffer setting on the server is a multiple of the block size of the database.

So if you use -B 1000 and the blocksize of the database is 2048, then the memory consumption is 1000*2048

This could have become a problem due to an incorrect Buffer (-B) or if someone has rebuilt/loaded the database with a larger block size.

Re: 100% Memory Usage

James R. Ferguson — Tue, 27 Aug 2002 18:45:45 GMT

Hi Jason:

One adjustment I'd make is to reduce the buffer cache ceiling ('dbc_max_pct') from 20% (which in your configuration translates to about 1GB RAM) to something like 5-8%. I'd set 'dbc_min_pct' to 2% at the same time. You should see a noticable improvement in memory utilization.

Regards!

...JRF...

Re: 100% Memory Usage

Jason Moorhead_2 — Wed, 28 Aug 2002 14:35:04 GMT

Thanks all. Lowering the dbc_min and max seemed to help quote a bit, as did turning off swapmem_on. We're still running high on utilization, but it looks like we're good now.

Without going with PerfView Alerts, is there a simple way to be alerted when memory utilization gets above a certain point? Does anybody run a cron script that e-mails when a certain level is reached?

Re: 100% Memory Usage

Patrick Wallek — Wed, 28 Aug 2002 15:54:23 GMT

I think you really want to turn swapmem_on back on!!!!!!

Otherwise with 5GB of RAM and only 2GB of swap, you could run into problems really fast! Since each process that starts has to reserve a bit of swap, once you reach the 2GB of swap reserved, you will NOT be able to start any more processes. If that happens I think you will get MANY complaints!

Another thing you could do if money allows is add more RAM to the machine. 5GB is not a whole lot of RAM anymore, and memory is getting pretty cheap.

Re: 100% Memory Usage

Jason Moorhead_2 — Thu, 29 Aug 2002 16:22:01 GMT

In the last day, I have made some changes that are helping. I've added another GB of RAM (6 total now), and dedicated a whole 9GB drive to swap. Things seem to be okay now, but I'd still like a simple way to monitor mem utilization without buying PerfView Alerts. Anybody have scripts already written?