topic Re: Oracle SGA with multiple shared mem segments in Operating System - HP-UX

Oracle SGA with multiple shared mem segments

Tim Nelson — Fri, 08 Jan 2010 17:07:32 GMT

rx7620
32GB RAM
HPUX 11.23
6/09 patch bundle

shmmax=28gb

The shared memory segments are being split. Typically we see a single segment via ipcs for oracle.

e.g.
sga=20gb
ipcs returns two 9gb and one 2gb segment

if set sga=10gb
ipcs shows two 4gb and one 2gb segment

Any ideas ?

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Fri, 08 Jan 2010 17:09:33 GMT

Oracle 10.2.0.3

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Fri, 08 Jan 2010 18:45:56 GMT

I believe I found what I needed.
Metalink ID 759565.1

and a good discussion here.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1353911

My rx7620 is a NUMA system and there is nothing wrong with the shared mem allocations.

(unless someone has more to discuss ;)

Re: Oracle SGA with multiple shared mem segments

Michael Steele_2 — Fri, 08 Jan 2010 23:18:14 GMT

Hi

a) did you turn off ccNUMA?
b) How did you turn ccNuma off?
c) Are you using Npars? Vpars? Dynamic Vpars? Other?
d) How are your db's and app's spread out?

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Sun, 10 Jan 2010 15:00:00 GMT

a) did not turn off NUMA
b) n/a
c) just a single nPar
d) Only on DB

Re: Oracle SGA with multiple shared mem segments

Duncan Edmonstone — Sun, 10 Jan 2010 15:48:17 GMT

Tim,

I doubt your SGA shared memory sizes are related to NUMA - more likely just down to your kernel parms related to shared memory. If you have metalink access have a read of article 15566.1 which explains how oracle will allocate shared memory segments.

HTH

Duncan

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Mon, 11 Jan 2010 15:30:14 GMT

I did check out the metalink article and it does look like oracle sees NUMA.

This and a ccNUMA video I watched does suggest to disable if we are in a flat config(1 nPar, no vPars, ILM) but also mentions to be cautious on disabling. There also is mention of increased benfits with 11iv3 but we cannot go there yet.

# kctune shmmax
Tunable Value Expression Changes
shmmax 28000000000 28000000000 Immed

# ipcs -ma|grep oracle
m 5734405 0x00000000 --rw-rw---- oracle dba oracle dba 52 9665806336 8418 8876 9:21:44 9:21:44 11:15:36
m 262152 0x00000000 --rw-rw---- oracle dba oracle dba 52 9680453632 8418 8876 9:21:44 9:21:44 11:15:36
m 229385 0x00000000 --rw-rw---- oracle dba oracle dba 52 2166444032 8418 8876 9:21:44 9:21:44 11:15:36
m 229386 0xd9edaca0 --rw-rw---- oracle dba oracle dba 52 45056 8418 8876 9:21:44 9:21:44 11:15:36

# mpsched -s
System Configuration
=====================

Locality Domain Count: 2
Processor Count : 8

Domain Processors
------ ----------
0 1 2 3 4
1 0 5 6 7

Re: Oracle SGA with multiple shared mem segments

Don Morris_1 — Mon, 11 Jan 2010 17:53:07 GMT

Might as well check the memory topology as well to see what Oracle sees.

Compile and run the attached with either +DD64 or +DD32 -D_PSTAT64 as you see fit and run with no arguments.

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Mon, 11 Jan 2010 17:59:50 GMT

Looks to be all interleaved (ILV) , as expected..

--- System wide locality info: ---
index ldom physid type total free used
0 0 0 CLM 0 0 0
1 1 1 CLM 0 0 0
2 -1 -1 ILV 31G 7496M 24G
----- ----- -----
31G 7496M 24G

I guess the question still exists is...

Is oracle seeing the system as NUMA type a bad thing, even though all the memory is interleaved ?

Re: Oracle SGA with multiple shared mem segments

Don Morris_1 — Mon, 11 Jan 2010 18:10:59 GMT

From a raw latency point of view, it shouldn't really matter -- Oracle can bind processes/threads to processors and craft "local" segments -- but the segment cost would be the same either way.

It could hurt you a little in that a single larger SGA has a better chance to form large pages and hence large translations -- but in your segment sizes, it wouldn't be too bad [the two 9Gb if really 9Gb and not "about" 9Gb would be 2 4Gb pages and 1 1Gb page at best, the 2Gb would be 2 1Gb pages] so I doubt you're getting a real increase in TLB miss rates either.

More likely is that a non-NUMA scheduling policy would be better -- there's really not much reason for any bindings, and there may well be scheduling hotspots you could avoid.

Re: Oracle SGA with multiple shared mem segments

Tim Nelson — Mon, 11 Jan 2010 18:27:29 GMT

Thanks Don,

Some of this is simply common sense but needs a dose of discussion to confirm.

Much appreciated.

Tim.

Re: Oracle SGA with multiple shared mem segments

Sameer_Nirmal — Mon, 11 Jan 2010 18:35:46 GMT

These 2 white papers are good references.

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-0547ENW.pdf

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-4194ENW.pdf