100% memory utilization

[Babu Yalamanchi]

Hi!

I have read all the issues posted related to memory and swap usage. They did not address my issue, so this post.

Here is my server configuration:
HP-UX 11.00 trusted
rp5430 2 CPUs
1250 MB RAM
2000 MB of swap space
Kernel parameters:
swapmemory_on =1
maxswapchunks=2048
swapchunk=2048
dbc_max=50
dbc_min=5
shmmax=1,073,741,824

This server runs Oracle DB 9i with ~530 MG SGA
This is a standby server with no users on
The vmstat returns the following:

procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
0 0 0 1546 22858 0 0 0 0

# swapinfo
The swapinfo returns the following:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 0 524288 0% 0 - 1 /dev/vg00/lvol2
reserve - 41508 -41508
memory 945664 945664 0 100%
0 0 0 485
547 83 0 0 100

A little bit history:
When I re-boot the server and bring up Oracle
the swapinfo shows the following:# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 0 524288 0% 0 - 1 /dev/vg00/lvol2
reserve - 36032 -36032
memory 945664 33224 912440 4%

We re-boot the Oracle DB from cron every night. After 2 nights we are running out of memory.

Can you help?
Thanks
Babu Yalamanchi

[James R. Ferguson]

Hi Babu:

You wrote that, "We re-boot the Oracle DB from cron every night".

Exactly what do you mean? Are you gracefully shutting down Oracle or are you doing a 'kill' of it.

If you are sending 'kill -9' then you are probably leaving shared memory segments all over and this is causing you to run out of memory.

Regards!

...JRF...

[Babu Yalamanchi]

..JRF...
Thx for quick response.

We 'dbshut' and 'dbstart' from cron using oracle as the owner.

[A. Clay Stephenson · Accepted solution]

You are reading something into the "memory" line of swapinfo that is not there. That does not mean that all of your memory is used. Note that your pageouts are zero. This is the real indication of swap usage. Swapinfo's "memory" line is the memory reserved by pseudoswap -- except that it really isn't reserved. You are running Oracle on a box with little memory and more physical memory would be better but as long as you keep the size of the SGA relatively small and the number of processes relatively small then you should be okay. The very first thing that that you should do is reduce your max_dbc_pct to no more than about 10% --- and setting a static buffer cache of about 100MB would be better still. You do this by setting bufpages to a non-zero value indicating the number of 4K pages (25600 = 100MB). 11.0 is not nearly as good as 11.11 in ramping down the dynamic buffer cache quickly. In your case, I would also reduce shmmax to about 600MB because on your box is someone does allocate a 1GB shared memory segment then you are really going to be in trouble. You also need to take care that multiple 500MB segments don't get allocated.

[James R. Ferguson]

Hi Babu:

Good. Well, I missed the fact too that you have your Unix buffer cache far too high for an Oracle environment. I'd reduce your 'dbc_max' to perhaps 5% and 'dbc_min' to about 2%.

With only 1.2GB RAM and an SGA of 0.5GB you don't have a great amount of memory. If performance is acceptable, then you're seeing about what you can expect.

Regards!

...JRF...

[Babu Yalamanchi]

Clay, James

I will combine ypur responses and try the
following config changes:
dbc_max=5
dbc_min=2
buffers=1992(I have this on another machine)

How do I avoid 'You also need to take care that multiple 500MB segments don't get allocated.' ? This certainly appears to be our case.
I will run this config a few times(days).

[A. Clay Stephenson]

No, you either use dynamic buffer cache by setting bufpages to a zero value or a static (fixed-size) buffer cache by setting bufpages to a non-zero value. In that case, the xxx_dbc_pct values are ignored. Your suggested value for buffers (sic) is much too small. Go with something near 100MB, a reasonable value for your box (bufpages=25600). I would at least use a 64MB buffer cache (bufpages=16384).

You prevent multiple 500MB shm segments by discipline. Simply don't allow it to happen. Reducing shmmax to 600MB reduces the ability of an overzealous DBA to grab all your resources but he can still make requests for multiple 600MB segments until virtual memory is exhausted. In your case, I would actually turnoff pseudoswap (swapmem_on=0) because it is really intended for use with machines that have much more memory than swapspace. It's actually safer for you to operate with swapmem=0.

[Simon Wickham_6]

Hi,

You need to check
dbc_min_pct
dbc_min_pct defines the minimum percentage of memory to be used by dynamic buffer cache. There can be performance problems from having dbc_max_pct different than dbc_min_pct. If dbc_max_pct and dbc_min_pct are too far apart problems can arise.

If you are running large processes you should consider using a fixed buffer cache size, and possibly setting the chatr +pd attribute (Large pages can increase the amount of RAM a program needs by rounding up to bigger pages, so don't go too far with it.) and dbc_min_pct 5 and dbc_max_pct 10 should be fine for this .

Regards,
Simon

[Babu Yalamanchi]

All who responded

Thank you all for your inputs. I have set the following parameters:
dbc_max=5
dbc_min=2
bufpages=16384(redundant)
swapmen_on=1

It works and is now stable. The solution has solved my problems.

Regards
Babu Yalamanchi

[Babu Yalamanchi]

This works!

[Ted Buis]

You might get even more stability, although peak activity would be slow, by increasing the device swap to a 4GB to give you even more virtual address space. When you run out of virtual address space, no more new processes can start, since there is no room to page out to and get more space in RAM to run the process.

[Babu Yalamanchi]

Ted
Thanks for the idea. I was keeping some device swap space for Oracle 10g we're expecting soon.

regards
Babu

[Babu Yalamanchi]

See previous