Memory usage 99%

[Rick Chen_1]

This is a two-server HP-UX RP4440 SG cluster. The secondary server's memory utilization stays 99% all the time while the primary only uses 70%. Both servers have the same Oracle database and applications running. Anyway to identify what is causing this problem?

#glance
CPU Util | 1% 1% 1%
Disk Util F FV V | 12% 12% 12%
Mem Util S SU UB B | 99% 99% 99%
Swap Util U UR R | 84% 84% 84%

[Mel Burslan · Accepted solution]

check to see if you have buffer cache set for the same percentage value on both primary and secondary. Excessive memory use, is usually caused by a system which was not tuned but left at the default kernel parameter settings. To check, run the command:

kmtune | grep dbc

on both servers and if the values on secondary shows up larger than the primary node (usually dbc_max_pct is set to 50), you found your culprit. If this is the case, you need to rebuild your kernel, which means down time, despite how short it is.

Hope this helps.

[Hein van den Heuvel]

>> Anyway to identify what is causing this problem?

Is it a problem?
99 is a pretty number!

Rick, the itrc formatting caused the relative sizes if "S SU UB B" to be lost.
If you reply, please replace spaces by "-" or 'retain formatting' or both.
Which hp-ux version?

Mel,
I respectfully disagree.
dbc_max_pct allows the cache to grow, if there is no contention for memory. This is goodness.

dbc_min_pct on the other hand might reserve memory which could be better used elsewhere.

As Rick is using a recent RP system, he might be using 11iV3 in that case the filecache_min / max params could be use, but possibly should not be used and defaults accepted.

See "man dbc_max_pct" --> obsoleted.

$ /usr/sbin/kctune | grep -E 'filecache|Changes'
Tunable Value Expression Changes
filecache_max 8160403456 Default Auto
filecache_min 816037888 Default Auto

fwiw,
Hein.

[Rick Chen_1]

This is quite interesting. The HP-UX version is 11.11. Both dbc_max_pct and dbc_min_pct are valid. comparing primary and secondary servers, the values are set different:
primary:
# kmtune | grep dbc
dbc_max_pct 5 - 5
dbc_min_pct 2 - 2

Secondary:
dbc_max_pct 50 - 50
dbc_min_pct 5 - 5

Is there a reason to set this way? If the primary server works fine with the current setting, is there a reason I should not set the same on the seconary?

Thanks

[Mel Burslan]

As Hein said, buffer cache grabbing is a courteous process and your system only grabs (upto 50% in your secondary server's case) the free memory if no other processes were asking for it. But it makes the memory utilization at or near 99-100% all the time, which in some cases makes the people wearing suits very nervous without a valid reason.

So, yes, you can make them the same, but if you are not experiencing a performance problem on your secondary server, do you really need to take the server down ? Of course if this is an idle member of the cluster, waiting for a failure to happen, i.e., there are no packages running on it, it is a different story.

But for the heart of the matter, yes you can do it and if you want to do it or not is upto you. All my 60+ hpux servers have these parameters tuned to 7-8% for the _max and 5% for the _min parameters.

[Rick Chen_1]

Mel,
Resetting these parameters resolved the problem. Why is it a problem? If one database is started up, the memory is already in 99% on the O/S, the other database instances will experience memory contention. Besides my Oracle 10g installation failed to link files due to OUT OF MEMORY. After reset the parameters, I rebooted the box and started up 3 databases along with associated applications, the memory usage is staying at 68%. Good it is my test environment so I can reboot it quickly.

Thank you all for your great help!

Rck

[Bill Hassell]

> failed to link files due to OUT OF MEMORY.

Actually, this is not a memory problem, it is a swap space problem. HP-UX is a virtual memory system so RAM is not the limiting factor, it is the RAM + swap space that limits your available space. You need to double of triple the total swap space. In glance, you can see this where swap space is 84% allocated.

The buffer cache is dynamic and even though the (bad) default of 50% was specified, this space should move down towards the min_pct value to make room for processes, albeit fairly slowly. And note that when you do indeed reach 100% RAM used, then paging starts and that can have a massive impact on performance. The best metric for low memory is the page out rate in Glance (or po column in vmstat).

[Rick Chen_1]

Thanks Bill for the comments. The only way I was able to complete Oracle 10g installation without the OUT OF MEMORY error is to shut down related Oracle applications and reduced the memory usage on the HP-UX server to 95%. Swap may play a role in the case, but I did not record what was the swap after I shut down the applications, with 99% of memory usage I had no way to push through the installation at the "linking stage".

[Bill Hassell]

I have heard that some versions of Oracle require 35 Gbytes of swap space in order to install and build the binaries. Once built, the swap space can be reduced to 5-10 GB and everything works fine. Why the Oracle installer is so piggy is a mystery...