Potential memory bottleneck with Oracle 9.2.0?

[Tom Krocak_1]

We have a rp5405 server with 4 CPs, 8 GB memory running HPUX 11.11, which is running Ross ERP using Oracle 9.2.0. Dynamic buffer cache via kernel params is set at 2-5%, and typcially is about 250MB; oracle filesystems are set to bypass buffer cache. There are 4 oracle instances with a total SGA of 5.0 GB. When the oracle DB connections are in the 150+ range, the memory utilization is near 100% and pageouts are in the 15-20/sec range. Processor usage is light, under 10%. When the DB connections are less than 150, memory usage is lower and somewhat proportional to the number of DB connections. For example at 60-70 connnections, memory is 85% utilized?

Are we memory bottlenecked at the 150+ DB connection level? Are DB connections ("ps -ef | grep LOCAL"), a good measure of oracle activity?

[Indira Aramandla]

Hi,

Memory management is one of the areas which has been given significant attention in Oracle9i. Oracle9i introduces a dynamic memory management feature which allows for dynamically resizing the buffer cache and shared pool. It also includes a buffer cache size advice mechanism that predicts the performance of running with different sizes for the buffer cache. Oracle9i also provides transparent management of working memory for SQL execution by self-tuning the initialization runtime parameters that control allocation of private memory. This feature helps low-end users to reduce the time and effort required to tune memory parameters for their data warehouse and reporting applications, while high-end users are able to avoid memory tuning for individual work loads.

One of the most common problems when running large numbers of concurrent users on UNIX machines is lack of memory. With Oracle9i, the dynamic SGA infrastructure will allow for the sizing of the Buffer Cache, Shared Pool and the Large Pool without having to shutdown the database.

In this new model, a new unit of allocation is created called the 'Granule'. A granule is a unit of contiguous virtual memory allocation. The size of a granule depends on the estimated total SGA size, whose calculation is based on the value of the parameter SGA_MAX_SIZE. This would be 4MB if the SGA size is less than 128MB, otherwise it will be 16MB.

The Buffer Cache, Shared Pool, Large Pool and Java Pool are allowed to grow and
shrink based on granule boundaries. SGA memory will be tracked in granules by SGA components. To monitor the creation, the V$BUFFER_POOL view can used.

The following are the Size parameters which define the sizes of the caches for buffers for the primary block size:

DB_CACHE_SIZE
DB_KEEP_CACHE_SIZE
DB_RECYCLE_CACHE_SIZE

In Oracle 9i it is possible to use the "Dynamic Buffer Cache Advisory feature" which enables and disables statistic gathering for predicting the behaviour of the buffer cache with different cache sizes.

Most UNIX systems today use a combination of paging and swapping to manage memory. Generally, you will see the following behavior:

System lightly used: no paging or swapping occurs. System under a medium load: paging occurs as RAM memory runs low System under a very heavy load: paging stops and swapping begins.

Make sure that the machine is not swapping at all and at worst paging lightly. This indicates a system with a healthy amount of memory available. To analyze paging and swapping, refer to the document.

Attached is document that puts together few aspects of Memory usage, IO and CPU and Oracle parameters inrelation to the above.

I hope this helps to some extent.


Indira A

[ericfjchen]

The better way you must do is to measure hit ratio of buffer and library cache. You can use Oracle statspack report. It will clearly show you the instance efficiency.

[Yogeeraj_1]

hi,

Me too would recommend to setup STATSPACK and run it on each system for 15 mins intervals during the problem period.

Statspack will help you identify the top sql, the big wait events, contention points, bad performance metric.

You can then compare the values from each. There could be application "problems" using one of the instances.

regards
Yogeeraj

[SteveKirby · Accepted solution]

The short answer is that you ARE memory bottlenecked.

As the other posters have pointed out, though, you have a lot of options to address this problem.

Some options:
- Get more RAM
- Resize the SGA's of the instances to something smaller. Under 9i this can be done dynamically as mentioned)
- connection pooling. It sounds like it is connections that are causing the problem... connection pooling might keep the memory load low enough to keep you happy.
- check disk i/o. (sar -d 5 5) or top (not as reliable, but easy) to look for i/o %

Questions: Are you running 64-bit or 32-bit... I think you will hit a 4GB limit if you are 32 bit that you wouldn't see unless the instances we active.

Steve

[Brian Crabtree]

Are you using PGA_AGGREGATE_TARGET as well? This will help define the pool size for that, however you will still get some overhead from the process level. To be honest, you have alot of databases for a limited amount of memory on the system. I would recommend going to 16g, or moving some of the databases off of the system.

Thanks,

Brian

[Tom Krocak_1]

Thnak you for your replies. I should note that I am a Unix SysAdmin and also do capacity management; I am not the DBA, but caught this issue as part of a routine monthly report, where I am putting together the first such report. We are using HP's PerfView agent and manager, and related metrics, which found these problems.

Indira - your comments were great, with a lot of good tips which I will make available to the DBA. Your note towards the bottom of the response on Unix behavior ("medium load: paging occurs as RAM runs low", etc), seems a little out of sync with Steve's comment which is definitive ("you are memory bottlenecked"). During the busy period, we have both page outs (15+/sec) and a swapout rate of 1/sec.

Ericfjchen and Yogeeraj - more good tuning stuff and will pass along to DBA.

Steve - appreciate the short and definitive answer ("memory bottlenecked") and the options, including the reference to the good comments from the DBA's, above. Relative to your comments/questions: disk busy as measured by "sar" and PerfView is low, in the 5-10% range; we are running 64-bit. Do you see a conflict with your answer and Indira's? (From his reference point, I think we have a "medium" load with paging, low memory and low level swapping/deactivations). I think we have a light CPU load (5-10%), light I/O load (max 5-10% LUN busy) and a heavy memory load, particularly at 150+ DB connections.

Thanks again, Tom

[Alzhy]

If Oracle is accepting direct connections (dedicated).. then your environment will have a set number of connections and each connection will have a set number of memory requirements.

You may want to suggest to your DBA to consider using MTS (multi-threaded server) to allow he instances to scale and better use memory...

[Adrian_72]

thank for all

[Steven E. Protter]

You might need to collect data on your system to get a better handle on your kernel and tuning situation. I'm attaching a script set that does that.

Also, here is a doc, written by an HP performance guru who picked up a software case I had with HP a ways back.

http://www1.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-682735245+1091122386701+28353475&searchCrit=allwords
Document id: UPERFKBAN00000726

SEP