topic Re: 100% memory utilization in Operating System - HP-UX

100% memory utilization

Babu Yalamanchi — Fri, 27 Jan 2006 10:56:34 GMT

Hi!

I have read all the issues posted related to memory and swap usage. They did not address my issue, so this post.

Here is my server configuration:
HP-UX 11.00 trusted
rp5430 2 CPUs
1250 MB RAM
2000 MB of swap space
Kernel parameters:
swapmemory_on =1
maxswapchunks=2048
swapchunk=2048
dbc_max=50
dbc_min=5
shmmax=1,073,741,824

This server runs Oracle DB 9i with ~530 MG SGA
This is a standby server with no users on
The vmstat returns the following:

procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
0 0 0 1546 22858 0 0 0 0

# swapinfo
The swapinfo returns the following:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 0 524288 0% 0 - 1 /dev/vg00/lvol2
reserve - 41508 -41508
memory 945664 945664 0 100%
0 0 0 485
547 83 0 0 100

A little bit history:
When I re-boot the server and bring up Oracle
the swapinfo shows the following:# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 0 524288 0% 0 - 1 /dev/vg00/lvol2
reserve - 36032 -36032
memory 945664 33224 912440 4%

We re-boot the Oracle DB from cron every night. After 2 nights we are running out of memory.

Can you help?
Thanks
Babu Yalamanchi

Re: 100% memory utilization

James R. Ferguson — Fri, 27 Jan 2006 11:04:39 GMT

Hi Babu:

You wrote that, "We re-boot the Oracle DB from cron every night".

Exactly what do you mean? Are you gracefully shutting down Oracle or are you doing a 'kill' of it.

If you are sending 'kill -9' then you are probably leaving shared memory segments all over and this is causing you to run out of memory.

Regards!

...JRF...

Re: 100% memory utilization

Babu Yalamanchi — Fri, 27 Jan 2006 11:17:19 GMT

..JRF...
Thx for quick response.

We 'dbshut' and 'dbstart' from cron using oracle as the owner.

Re: 100% memory utilization

A. Clay Stephenson — Fri, 27 Jan 2006 11:18:03 GMT

You are reading something into the "memory" line of swapinfo that is not there. That does not mean that all of your memory is used. Note that your pageouts are zero. This is the real indication of swap usage. Swapinfo's "memory" line is the memory reserved by pseudoswap -- except that it really isn't reserved. You are running Oracle on a box with little memory and more physical memory would be better but as long as you keep the size of the SGA relatively small and the number of processes relatively small then you should be okay. The very first thing that that you should do is reduce your max_dbc_pct to no more than about 10% --- and setting a static buffer cache of about 100MB would be better still. You do this by setting bufpages to a non-zero value indicating the number of 4K pages (25600 = 100MB). 11.0 is not nearly as good as 11.11 in ramping down the dynamic buffer cache quickly. In your case, I would also reduce shmmax to about 600MB because on your box is someone does allocate a 1GB shared memory segment then you are really going to be in trouble. You also need to take care that multiple 500MB segments don't get allocated.

Re: 100% memory utilization

James R. Ferguson — Fri, 27 Jan 2006 11:22:39 GMT

Hi Babu:

Good. Well, I missed the fact too that you have your Unix buffer cache far too high for an Oracle environment. I'd reduce your 'dbc_max' to perhaps 5% and 'dbc_min' to about 2%.

With only 1.2GB RAM and an SGA of 0.5GB you don't have a great amount of memory. If performance is acceptable, then you're seeing about what you can expect.

Regards!

...JRF...

Re: 100% memory utilization

Babu Yalamanchi — Fri, 27 Jan 2006 12:10:28 GMT

Clay, James

I will combine ypur responses and try the
following config changes:
dbc_max=5
dbc_min=2
buffers=1992(I have this on another machine)

How do I avoid 'You also need to take care that multiple 500MB segments don't get allocated.' ? This certainly appears to be our case.
I will run this config a few times(days).

Re: 100% memory utilization

A. Clay Stephenson — Fri, 27 Jan 2006 12:19:41 GMT

No, you either use dynamic buffer cache by setting bufpages to a zero value or a static (fixed-size) buffer cache by setting bufpages to a non-zero value. In that case, the xxx_dbc_pct values are ignored. Your suggested value for buffers (sic) is much too small. Go with something near 100MB, a reasonable value for your box (bufpages=25600). I would at least use a 64MB buffer cache (bufpages=16384).

You prevent multiple 500MB shm segments by discipline. Simply don't allow it to happen. Reducing shmmax to 600MB reduces the ability of an overzealous DBA to grab all your resources but he can still make requests for multiple 600MB segments until virtual memory is exhausted. In your case, I would actually turnoff pseudoswap (swapmem_on=0) because it is really intended for use with machines that have much more memory than swapspace. It's actually safer for you to operate with swapmem=0.

Re: 100% memory utilization

Simon Wickham_6 — Fri, 27 Jan 2006 12:29:40 GMT

Hi,

You need to check
dbc_min_pct
dbc_min_pct defines the minimum percentage of memory to be used by dynamic buffer cache. There can be performance problems from having dbc_max_pct different than dbc_min_pct. If dbc_max_pct and dbc_min_pct are too far apart problems can arise.

If you are running large processes you should consider using a fixed buffer cache size, and possibly setting the chatr +pd attribute (Large pages can increase the amount of RAM a program needs by rounding up to bigger pages, so don't go too far with it.) and dbc_min_pct 5 and dbc_max_pct 10 should be fine for this .

Regards,
Simon

Re: 100% memory utilization

Babu Yalamanchi — Thu, 02 Feb 2006 11:26:53 GMT

All who responded

Thank you all for your inputs. I have set the following parameters:
dbc_max=5
dbc_min=2
bufpages=16384(redundant)
swapmen_on=1

It works and is now stable. The solution has solved my problems.

Regards
Babu Yalamanchi

Re: 100% memory utilization

Babu Yalamanchi — Thu, 02 Feb 2006 11:28:18 GMT

This works!

Re: 100% memory utilization

Ted Buis — Fri, 03 Feb 2006 22:35:38 GMT

You might get even more stability, although peak activity would be slow, by increasing the device swap to a 4GB to give you even more virtual address space. When you run out of virtual address space, no more new processes can start, since there is no room to page out to and get more space in RAM to run the process.

Re: 100% memory utilization

Babu Yalamanchi — Mon, 06 Feb 2006 08:10:19 GMT

Ted
Thanks for the idea. I was keeping some device swap space for Oracle 10g we're expecting soon.

regards
Babu

Re: 100% memory utilization

Babu Yalamanchi — Mon, 06 Feb 2006 08:13:27 GMT

See previous