Memory utilization and tunning in TRU64

[Martin Wolff]

Hi,

I am running an aplication that uses Oracle 8 at a DS-25 cluster running TRU64 5.1B PK4.

Oracle was running out of shared memory so DBAs changed a Oracle parameter in order to let Oracle get more memory. This made the system run out of memory, and Oracle could not start again.

Before telling DBA´s to make the change i checked the output of vmstat -P.

I was told long ago that to check the memory that could be used(free), i should sum the free-pages and the inactive-pages. Is this correct?

Also i have seen that UBC is getting a lot of memory, how can i check if it´s actually using it?.

I know i can reduce the max percentage UBC can grow, but i want to be sure if i will not trigger another issue by doing this.

Thank you very much in advance,

Kind regards,
Martin.

[John Manger · Accepted solution]

I think you mean, the DBAs changed an Oracle parameter and now Oracle can't start ... because its asking for more than the system can give.

What did they change ? If you tell the forum what they changed, we may be able to suggest what other things may need adjusting.

Altering the SGA (?) without considering other parameters is not a good way to proceed ;-)

You may also want to post your /etc/sysconfigtab, and whatever console messages and/or Oracle log messages appeared when Oracle couldn't start.

JM

[Martin Wolff]

Yes that's what i meant. They did that change because they were having problems.
I will gather precise information on the errors and changes.

My goal is to know how memory works at TRU64, and know if i can tune any parameter to optimize memory utilzation, and make the system survive some more years.

Thank you very for your help!

[Martin Wolff]

The problems seen by the people that works with the Oracle 8.1.7.4 database is that the shared-pool was totally consumed.
After changing the shared pool from 50MB to 150MB, the database startup showed:

SVRMGR> ORA-27102: out of memory
Compaq Tru64 UNIX Error: 12: Not enough space
Additional information: 1
Additional information: 514

Before doing this we checked the vmstat -P results and the free-pages + inactive pages ~ 450MB, so i told them to proceed to increase the shared-pool in 100MB. Perhaps that was unwise, but the errors in the database access were preventing, and are preventing the systema to work correctly.

I will post the actual vmstat -P output and the /etc/sysconfig file in the next posts

[Martin Wolff]

ro:/>#vmstat -P

Total Physical Memory = 2560.00 M
= 327680 pages

Physical Memory Clusters:

start_pfn end_pfn type size_pages / size_bytes
0 404 pal 404 / 3.16M
404 327671 os 327267 / 2556.77M
327671 327680 pal 9 / 72.00k

Physical Memory Use:

start_pfn end_pfn type size_pages / size_bytes
404 521 scavenge 117 / 936.00k
521 1681 text 1160 / 9.06M
1681 1939 data 258 / 2.02M
1939 2437 bss 498 / 3.89M
2437 2729 kdebug 292 / 2.28M
2729 2737 cfgmgmt 8 / 64.00k
2737 2739 locks 2 / 16.00k
2739 2755 pmap 16 / 128.00k
2755 4078 unixtable 1323 / 10.34M
4078 4090 logs 12 / 96.00k
4090 11752 vmtables 7662 / 59.86M
11752 327671 managed 315919 / 2468.12M
============================
Total Physical Memory Use: 327267 / 2556.77M

Managed Pages Break Down:

free pages = 3073
active pages = 103235
inactive pages = 57864
wired pages = 62164
ubc pages = 89699
==================
Total = 316035

WIRED Pages Break Down:

vm wired pages = 9432
ubc wired pages = 0
meta data pages = 9747
malloc pages = 36616
contig pages = 3702
user ptepages = 2050
kernel ptepages = 330
free ptepages = 8
==================
Total = 61885

[Martin Wolff]

I am attaching the /etc/sysconfigtab file and also got more precise information about Oracle errors seen before the change.

Precise errors:

ORA-04031: unable to allocate 4248 bytes of shared memory

Then DBAs changed the following parameter:

shared_pool_size = 50M --> 100M

[Pieter 't Hart]

Martin, check this :
"Tuning HP Tru64 UNIX V5.1A and V5.1B for Oracle Environments"

http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA0-9949ENW.pdf

[Martin Wolff]

A collegue has noted the following in the proc subsystem:

max_per_proc_address_space and per_proc_address_space parameters are less than default,

max_per_proc_address_space = 536870912
per_proc_address_space = 536870912

Can this prevent Oracle from taking the space it´s needing?

Thanks in advance,
Kind regards,
Martin.

[Pieter 't Hart]

It could just be your DBS made a mistake about the units of memory allocation.
It could be bytes, 512byte blocks, 2Kbyte pages or even 8Kbyte pages.

if your DBA meant bytes, but the parameter adresses blocks or pages, the result will be obvious.

You could also generate a sys_check output?
It will advise you about current environment and advise system parameters.
It my even recognize the Oracle environment and advise accordingly.

I know many administrators want to be in control themselves, but ist's a helpfull tool.
Especially it helps checking relationship between parameters.
you'll allso find many statements with options there you can use separately in the future.
If needed use the man pages for explanation.

[Martin Wolff]

Tonight i will modify:

max_per_proc_address_space to 4294967296.
and
per_proc_address_space to 4294967296

And i will run the sys_check utility, as i started it at 15:30hs and my system CPU utilization increased too much for a busy hour.

I will check the output of sys_check, but is there any way you can tell me if Oracle is getting trouble to allocate memory at startup because of this parameters or any other Kernel parameter?

In your experience, what Kernel params can be producing the Oracle

SVRMGR> ORA-27102: out of memory
Compaq Tru64 UNIX Error: 12: Not enough space

?

[Pieter 't Hart]

regarding
SVRMGR> ORA-27102: out of memory
Compaq Tru64 UNIX Error: 12: Not enough space
this link http://www.itags.org/database/234237/ suggests it is a swapspace problem.
Swapspace size is normally sized about 1,5 times physical memory.
Offcourse adding swapspace (virtual memory) will reduce "out of memory" messages.

=========================
But on a databassystems normally you don't want swapping to occur!
=========================
Swapping in/out reduces performance.
So the total memory requested by oracle should be less than max physical memory minus room for other processes including users and batchprocesses.
As you report "Total Physical Memory = 2560.00 M" I would suggest to limit the oracle parameters to use max 2GB instead of raising the system limit so swapping will occur.
If your database really needs more memory, you better add physical memory before increasing oracle parameters.



regarding :
ORA-04031: unable to allocate 4248 bytes of shared memory
you need to look also at the shm_max

from the link i posted earlier.
If only Oracle is being used on the system, you can increase the size of shm_max to 4 GB minus 16
MB, that is:
0xff000000 or 4278190080

This should better address your initial problem then increasing oracle using more memory.

[Martin Wolff]

Thank you Pieter,

Regarding swap space i think i am ok (as seen in attached file).

Actually i have done a maint operation this morning. I changed max_per_proc_address_space to 4294967296 and per_proc_address_space to 4294967296.

After that i changed oracle init.ora shared_memory_size to 150M and the database restarted fine without problema.

Now a few hours later, the main aplication running in the server started telling it was out of memory, and stopped working correctly.

Shared memory is set to 2139095040 ~ 2GB so i guess that´s not a problem also.

We went back those two parameters to the original values 536870912, and the shared_pool_size back to 100MB and the system went back to normal.

I checked vmstat -P and there were still free pages left at the moment of the failure, so i am confused and in a big trouble!!

Any further help will be highly appreciated,
Kind regards,
Martin.

[Pieter 't Hart]

In my opinion 8GB swapspace is extremely large for a system with 2,5G physical memory!
Especially in a database environment.
How many "normal" (non-database) processes are ther on youtr system?

I think you and your DBA need to review the oracle parameters against unix system parameters.

maybe you can post both sysconfigtab and oracle config to review?

My knowledge is from a sybase environment, not oracle.
But maybe i can find some mismatch.

[Martin Wolff]

Hi Pieter,

There are ~20 non database proceses.

By the way, i made a terrible mistake when changing the max_per_proc_address_space to 4294967296 and per_proc_address_space to 4294967296.

I did the operation with:
"sysconfigdb -u -f stanza-file proc", i put a u!!! instead of m!!! So all proc parameters except the ones i put in the stanza file were set to very low values(defaults?)

I have changed them to the values in the attached file and now the system is working for 8 hours without any proccess complaining about lack of memory.

I still do not know what was the Kernel parameter that made the aplications fail to get memory, but for sure i made a terrible mistake.

Thank you very much Pieter,
Kind regards,
Martin.

[Martin Wolff]

I have attached the memory usage of one of the machines with trouble, it looks ok as far as i understand.

[Pieter 't Hart]

Hi Martin,

sysconfigdb should make a (numebered) backup before applying changes.
Have you checked if there are any sysconfigtab. files present?

If so you may be able to retrieve the missing sysconfig parameters.