Serious Memory Problem

[dicky_1]

Hi there,

I have two nodes of HP-UX Itamium with 4G memory and 4G swap space. Both configuration is identical. The following is the output of "swapinfo -ta" command for each node:

Node A (Normal)
===============
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 1156456 3037848 28% 0 - 1 /dev/vg00/lvol2
reserve - 3037848 -3037848
memory 4182352 2238588 1943764 54%
total 8376656 6432892 1943764 77% - 0 -

Node B (Abnormal)
=================
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 275068 3919236 7% 0 - 1 /dev/vg00/lvol2
reserve - 3639516 -3639516
total 4194304 3914584 279720 93% - 0 -

As you can see, the "Memory" row of Node B is missing and the total of memory used is very high even though Node A is serving much more Oracle connections.

The situation is worse off when Oracle requires more memory. Either the Oracle instance or the node itself will go crash. One more strange thing is the usage of "dev" is very low (no more than 20% at most) even the "total" is close to 100%.

Please. shed me light on that because the nodes are on production! Million Thanks!

Dicky

[Patrick Wallek]

The memory line is missing on node b because you do not have swapmem_on enabled (set to 1) in the kernel.

That being said, it appears to me that you REALLY REALLY REALLY need to add more RAM to these machines. 28% swap usage is NOT GOOD.

4GB of RAM is not much in modern machines. I wouldn't buy an HP-UX machine with less than 8GB if it is going to run Oracle. I would probably consider 16GB or more depending on the number of users accessing the DB.

[Reshma Malusare]

Hi Dicky,
I am agree with Patrick.If you are working with oracle then you always prefer higher memory as such applications take lots of memory & with low memory it will cause performance degradation.

Thanks & Regards
Reshma

[A. Clay Stephenson]

Moreover, in your case, setting swapmem_on=1 really does nothing for you. It's really intended for those systems which have more memory than swap --- yes, you heard that correctly. This is not true of your system so at most, swapmem_on=1 simply buys you a little more of a false sense of security. 4GiB of physical memory to run Oracle is consided tiny these days. You can prevent the crashes by adding more (real) swap space because your system is not limited by the amount of physical memory but rather the amount of virtual memory --- but your performance is going to be terrible. If you have been doing some kill -9's, it is quite possible you have large chunks of shared memory that were never freed.

[dicky_1]

Thanks for all your suggestions! Just curious to know why the "DEV" used % is very low while the "TOTAL" used is 99% or even 100%? Because i'm newbie in HP-UX, can you all please guide me how many percentage of "DEV" used in normal?

Btw the crash symptom (out of memory) just happened on Node B but not on Node A. Does anyone has encountered such an experience?

Thanks in advance!

Dicky

[Reshma Malusare]

Hi Dicky,
Welcome to HP-UX forums.
Please also read:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33 on how to reward any useful answers given to your questions.

[Hein van den Heuvel]

What they are all saying is that you may have over commited your memory. So you need to review that. Speficially you want to see how much SGA each Oracle instance is allowed to grab. With that gone, and DBC_MIN gone, and some kernel space gone, how much memory is left? How does that compare to the PGA_AGGREGATE_TARGET for your Oracle instances? how much memory per connection is avaible?

You may have to REDUCE the SGA to get more stability at the cost of more IO?
You may have to TUNE the SGA at no cost other than brain power. Does the SGA have one of those 250mb Java pool which are rarely or not used in some cases? Trim that!
Multiple pools? MRU limits?

If agree that we now often need multi-GB sized SGAs for Oracle instances, but some Oracle instances with tightly controlled resources are perfectly happy with 250MB or less. You woudl need to work at that though. Throwing memory at it may be more cost effective.

Regards

Hein van den Heuvel
HvdH Performance Consulting

[Patrick Wallek]

How much dev usage is normal? I strive for 0% dev usage.

I absolutely do NOT want to page anything out, if possible. Paging out reduces system performance severely.

[RolandH]

Hi dicky,

welcome to the forum.

check these two kernel parameters
dbc_max_pct
dbc_min_pct

with kctune

Oracle has it's own chaching/buffering mechanisem (SGA) and does not need the dynamic buffer cache of the OS.

If these values are more than 10% in a 4GB System, which is for Oracle, I recommend to set both values to 10%.

Check a kernel parameter with kctune i.e.
kctune dbc_max_pct

Change a value with kctune i.e.
kctune dbc_mac_pct=10

Change a kernel paramter to default i.e.
kctune dbc_max_pct=

Check all kernel parameter
kctune

Regards
Roland