90% average mem utilization 24x7

[Jeremy Loukinas_1]

I have a server running HPUX 10.20. Runs Informix and a couple other applications.

Here is my problem. My metrics are coming back every month with average memory utilization of 90% + at all times.

Can anyone give me some hints or suggestions on what to look at first?

Jeremy

[Sridhar Bhaskarla · Accepted solution]

Hi Jeremy,

Look for dbc_max_pct and dbc_min_pct values. By default, they are set to 50%.

sysdef |grep dbc

And the values you get are %'s of your physical memory. Decrease them to get around 300-400MB.

-Sri

[John Poff]

Hi,

First thing I would look at is your buffer cache. What are your dbc_min_pct and dbc_max_pct kernel parameters set for? How much physical memory does the system have? If you have dbc_max_pct set to the default of 50%, you can drop it down to around 8-10% and pick up some extra memory.

JP

[Patrick Wallek]

You're thinking 90% is bad? If it hovers around 90%, that is pretty good. The biggest indicator that you are having memory problems is if you start paging. Keep an eye on your 'swapinfo' output. If you never start paging, and your memory usage stays around 90%, then you have no problems.

I wouldn't really worry unless your memory usage goes up to and stays at 100%. As it is, you are using most of your RAM, which is a good thing.

[Ken Hubnik_2]

That's great!!! You are getting your moneys worth out of your memory. As long as you are not paging tremendously or having pages deactivated then you are fine.

[Jeremy Loukinas_1]

Here is my swapinfo output.

hp39:/ # swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 393216 28920 364296 7% 0 - 1 /dev/vg00/lvol2
reserve - 73264 -73264
memory 91656 64628 27028 71%
hp39:/ #

[Sridhar Bhaskarla]

Hi,

Well, I will not be too excited if my memory hits above 90% on production systems as anytime I may get hit hard with extra load. I am happy as long as my memory util is around 80%.

Looks like you did swap a bit around 29MB of it is put on the device swap. It does tell that you are almost on the edge. But this is not a worrysome figure right now.

Check for dbc* parameters as indicated above and reduce them if you are using them with default values. You will get some memory freed up.

Also, look at your memory trend. If it gradually increasing over a period of time, then there is a chance that you have memory leaking processes. Generate regular outputs of

UNIX95= ps -e -o "vsz args" |sort
^ note the space

and see if the first column for any of the processes is growing over the time. That may be an indication of leaks.

-Sri

[Jeremy Loukinas_1]

Looks like I am using 7% of my swap.. Am I reading this correctly?
I formatted this in mb instead of kb.

hp39:/ # swapinfo -m
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 384 28 356 7% 0 - 1 /dev/vg00/lvol2
reserve - 71 -71
memory 90 64 26 71%
hp39:/ #

[Jeremy Loukinas_1]

what are the kernel params

dbc_min
dbc_max

?

They are set to the defaults of

5 and 50.

Jeremy

[Sridhar Bhaskarla]

Oops my ^ shifted place after posting.

The space is after UNIX95=

-Sri

[Sridhar Bhaskarla]

Hi Jeremy,

After looking at your swapinfo output, you do not seem to have more than 200MB of physical memory. So my initial statement to adjust dbc_max_pct to get 300MB would not be valid.

Make dbc_max_pct to 15% and dbc_min_pct 5% and see how your application works. That should release quite some memory.

-Sri

PS: dbc* are parameters for data buffer cache taken out of physical memory that sits in between user memory and the physical storage as a temporary storage. It buffers IO so the most recent reads do not have to go to the disk again and multiple write IOs can be combined if possible to reduce IO overhead.

[Frank Slootweg]

*Where*, i.e. which utility, which parameter, etc., do you see "90% average mem utilization 24x7".

If it is really memory, i.e. RAM utilization, i.e. like "Mem Util" in Glance, then, as others indicate, 90% is *good*, not bad.

On HP-UX, main memory works like a cache and a good cache is always full. In order to allow fast startup of samll processes, HP-UX keeps memory nearly full, normally around 95%. If you see memory utilization of 97% and higher or/and paging/swapping, *then* memory pressure is to high.

Re: "swapinfo -m" output:

> Looks like I am using 7% of my swap.. Am I reading this correctly?

No, you should use the "-t" option for swapinfo and only look at the "total" line, *there* you will probably see a much higher USED percentage. The other lines are correct, but often only create confusion. See the manual page for details, but first take an asperin! :-)

[Pete Randall]

Jeremy,

For an explanation of kernel memory parameters (dbc_min, dbc_max, etc.), see this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x89a950dde50cd71190050090279cd0f9,00.html


Pete