Performance - Confirm Need for more Memory

[Greg Stark_1]

We have a 2-way, 440 L2000 with 1GB of RAM and 2GB of swap running 11.00. The server is mainly used to host Oracle apps. Lately, the system performance has seem to get worse as new components/functionality has been added to the apps.

After reading many of the posts regarding memory management, I've come to the conclusion that I need more physical memory. Before spending the $, I thought I'd ask some of you to confirm based on the following:

1. The vhand process is running quite often and is often one of the top CPU hogs according to top and "ps -ef |sort +3".

2. swapinfo -tm is as follows
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 377 647 37% 0 - 1 dev/vg00/lvol2
dev 1024 381 643 37% 0 - 1 /dev/vg01/swap1
reserve - 240 -240
memory 721 632 89 88%
total 2769 1630 1139 59% -0 -


3. vmstat often shows po's above ten, sometimes between 30 and 50.

Given this, do you also think increasing my RAM to 2 or 3 GB's would fix the problem? Is there any other commands I sure run to confirm a memory shortage(I do not have Glance)? Application performance also seems to get really bad when the omniback backup runs. Could this just be because of the added io/cpu load omniback consumes?

Thanks again,
Greg

[John Dvorchak · Accepted solution]

I agree with you ascertion that you are out of memory. Vhand is of course a big hint but the swapinfo paints a pretty ugly picture of your system. Also you have noted that as time has gone on and more and more applications, features etc. are added the performance is getting worse.

Without glance it is pretty tough to make a guess as to what/who is using all of your RAM but we do know that Oracle will use a big chunk.

[Sridhar Bhaskarla]

Hi Greg,

First check your buffer cache parameter.

kmtune -l -q dbc_max_pct
kmtune -l -q nbuf
kmtune -l -q bufpages

If nbuf and bufpages are both set to 0, then you are using dynamic buffer cache. So if dbc_max_pct is set to 50% which is default, lower the setting immediately back to get around 300-400MB and adjust dbc_min_pct to around 100MB.

If it is low, then you are running short on memory resources. You have around 750MB sitting on the swap. So, the minimum you would add will be 1GB. Also, if you do not have much data in vg01, then make the priority of swap1 to 0 so that that area will be accessed first. You will not hit the root disk for swap activity.

-Sri

[James Murtagh]

Hi Greg,

Yeah, from what you have said more memory will help a lot. From the swapinfo output approximately 760MB is being paged out during normal operation. Therefor increasing memory by about a GB will enable active processes to run in-core. Another useful field from vmstat is the "free" column which is the free memory in 4096 byte pages, however the swapinfo in this case is the most use.

Omniback will need to read the files into memory before writing to tape, so it is probably down to the same issue I would have thought.

The decision to go to 2GB or 3GB should depend on the maximum memory load on the server, as you mentioned probably when Omniback is running. However, take into account the page out rate as this will skew the picture some what - I would estimate on the high side to be safe.

Regards,

James.

[John Dvorchak]

I got this link from the ITRC the other day. Maybe it can help also:

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&admit=-682735245+1048202796678+28353475&docId=200000063203050

[Michael Tully]

Your assumption is correct, you need more RAM.
There are a few things you can look at as well, although I think having only 1Gb for an Oracle database is extremely small. I would look to increase to at least 4Gb. RAM is dirt cheap anyway. Also look at the amount of buffer cache you currently have, you may have it set too high anyway.

# kmtune -l -q dbc_max_pct
# kmtune -l -q dbc_min_pct
# kmtune -l -q nbuf

I would also look to increasing your existing swap space and keep it 2xRAM=swap.

[Patrick Wallek]

Yes, I would definitely add more RAM. I'd take it up to at least 4GB.

If you are leary of paying HP's prices for memory check out Kingston.

http://www.ec.kingston.com/ecom/configurator/modelsinfo.asp?SysID=+8813+&distributor=0&submit1=Search

[A. Clay Stephenson]

The significant pageout rates are your real key. You are definitely swapping. These days, even modest Oracle instances can easily require 2GB - not counting the applications. The idea is to not swap at all because as soon as you do, you take a huge performance hit. I am definitely not in the 2x swapspace camp - that rule is about 1o years old. If you do have more swap than memory then turn off pseudoswap - it's just one more bit of overhead that you don't need.

Ideally, configure just a small amount of primary swap (512MB or so), enable pseudoswap, and add enough additional swap so that you can access all your virtual memory. You really can't talk about performance and swap in the same sentence anymore.

[Steven E. Protter]

You swap use seems a bit high, and paging indicates memory is short.

HP-UX 11.11(11i) uses more memory just to run than prior versions, thereby raising overall memory requirements.

I am attaching my production performance data collection script which does not require glance. Note you can use glance on a 30 day trial to try and help you with this issue.

SEP

[Frank Slootweg]

(If you do not have Glance,) You can also look at the "free" number in the vmstat(1) output. As the manual page says, that is the size of the free list (in pages, of 4KByte/page). If the free memory gets substantially below 5% of RAM, i.e. (substantially) more than 95% is used, then the memory pressure is too high and paging or even deactivation/re-activation will start.

Note that 5%/95% free/used is perfectly fine as HP-UX treats main memory as a kind of cache and a good cache is always full. HP-UX tries to keep it nearly full instead of totall full, in order to speed up the start up of additional 'small' processes.

[John Bolene]

For our Oracle baxes, we use dbc_max_pct = 5, most of the I/O is all database so not much need for buffer space.

We run with 1G/cpu on the K and L machines (most are 4 way) and 2G/cpu on the N (most are 6 or 8 way) and superdomes (the big mama with each vpar at 16 way).

Memory has come down in price quite a bit and you can't have enough real memory for Oracle.

That high vhand means that memory is tight and it is having to constantly steal back pages from applications to give to other running processes.