shmmax with Oracle Applications on hp-ux 11.11.

[Stuart Abramson_2]

I just read a ticket on shmmax with 32-bit oracle applications.

What should I set shmmax to if I'm running full 64-bit Oracle Applications 11i with Oracle 9i on hp-ux 11.11 with 12 GB RAM and 24 GB swap?

Someone suggested full memory + swap. In my case that would be 36 GB!

[Tomek Gryszkiewicz]

On my Oracle box (20 procs, 40GB) we set it to about 17GB - this value was suggested by one of the "Oracloids" :-)

-Tomek

[Alzhy · Accepted solution]

That would be correct. Although I doubt any DBA would actually size an Oracle instance to have an SGA (which takes 1 shared memory segment chunk)that will exceed available RAM.

For 32-Bit applications, it really does not matter but most Oracle documentation would recommend setting it to 4 GB (max address space for 32-bit apps?) regardless of memory installed .. Of course rarely these dyas you will find a DB server whose total VM is less than 4 GB anyway.

If you will be having multiple DB instances, make sure the total SGA sizes will not exceed installed RAM or encroach into swap area(s) -- use "swapinfo -atm". If you see your swap devices being used already -- chances are your total sum of your SHMEM segments (SGA mostly) have exceeded RAM and you're most likely be memory bound already.

[Stuart Abramson_2]

Did they have some kind of formula?

[Stuart Abramson_2]

Nelson, you said: "Although I doubt any DBA would actually size an Oracle instance to have an SGA (which takes 1 shared memory segment chunk)that will exceed available RAM."

Then:
1. If my RAM in 12 GB, but I use some of it for kernel space, process space, and buffer cache, I want to set my "shmmax" to, let's say, 8 GB.

2. If I have two (equal size) SIDs on the same machine, I would then set "shmmax" to 4 GB.

Does that sound right?

[Jeff Schussele]

Hi Stuart,

I would find out what size SGA the DBAs are using & that would be the absolute minimum.
Depending on what that size is, throw extra in to cover any non-Oracle needs.

I've just heard from very reliable sources that Oracle 9i has some new tricks up it's sleeve that can really cause problems if you're not paying close attention
For EX:
Let's say the DBAs have set up a 10GB SGA & shmmax is set to the default 1GB. Oracle 9i will now - instead of erroring out - go ahead & create 10 1GB shmem segments & the resultant performance hit is devastating.

So consult with the DBAs on this & throw in extra for other overhead.

HTH,
Jeff

[Alzhy]

Stuart,
Bear in mind that SHMMAX is not immediately allocated on boot time. It is just a guidance as to how large of a Shared Memory Segment can the kernel allow to be created. You can even declare it to be well larger than your VM (swap + memory) and the kernel will not complain. If I were you, the safest will be to set it to the size of your RAM. FOr some breathing room -- you could add half of your swap -- so 24 GB total SHMMAX.

Again, SHMMAX is not allocated at boot time. It is just a guidance.

[Jean-Luc Oudart]

As mentioned, SHMMAX is a high water mark and the SGA should be limited by this value.

on 9i, the recommended size is same as available memory on the server.

source =metalink doc id 169706.1
Rgds,
JL

[Jeff Schussele]

Hi Nelson/JL,

I don't agree that it should be set to those high values. I view shmmax the same as the maxXsiz parms - they're fences - fences to keep processes/application from monopolizing or exhausting system resources. In this case RAM. And they should be set to reasonable values agreed to by all parties & then all must be required to adhere to them.
Applications are like buffer cache in a sense, they're going to use all you allow them to & if you give them too much - they'll use it & the resultant consequences may not be desirable.

My 2 cents,
Jeff

[Hein van den Heuvel]

Jeff wrote: "Let's say the DBAs have set up a 10GB SGA & shmmax is set to the default 1GB. Oracle 9i will now - instead of erroring out - go ahead & create 10 1GB shmem segments & the resultant performance hit is devastating."

Other replies imply this.

Balloney. You guys are seeing ghosts.
I challenge you to actually measure the difference between 1*10GB and 10*1GB.
Now if you made that 1000*10mb then that woudl matter, but as long as you have picked a reasonable value ( 500MB - 4GB ) then you will not be able to tell teh difference. Just let Oracle create multiple segments. No harm done and be more assured that an other application will not go too wild.

The suggestion to set shmmax to physical memory + swap borders on the insane!

hth,
Hein.

[Alzhy]

Agreed Hein, I actually never have seen any actual metrics as to how Oracle performs when SGA is in just 1 SHM segment or some "saner" number of segments. The suggestion to set SHMMAX to as large as VM is more largely a safety net for malformed and misbehaving applications (and instances?)so we as admins can analyze effectively during such situations. My rule of thumb is to always ask the DBA's what biggest SGA is expected on a system and set it to that value. We also set a minimum of 1. But I guess with 11i with cooked Filesystems - the norm these days is to always have the biggest SGA that can possiby be allocated for an instance. I leave it out the the HPUX doctorates and laureates to have a more exhaustive assessment of which is correct.

[Jeff Schussele]

Hi Hein,

With all due respect I believe you're wrong.
Do you have any idea the costs associated with the shmset(), shmget() & shmat() system calls? Simply imagine a process needs to attach to 10 shmem segments & another needs to only attach to one. Which one do you think will be faster? Then the overhead associated with working down through process structures that are attached to 10 separate shared memory segments as opposed to only one. Again which one will be doing more work in your opinion? And remember this will all be System CPU time which will be stolen from the User CPU time.

I really think you'd benefit from attending one of the great HP classes like HP-UX Internals or even the Performance & Tuning Class.

And here's a great paper from Stephen Ciullo that is very informative:

http://h21007.www2.hp.com/dspp/ddl/ddl_Download_File_TRX/1,1249,280,00.pdf

Rgds,
Jeff

[Jean-Luc Oudart]

Jeff,

as I said this a high water mark only.

The right process should be proper planning and communication between Sysadmin and DBA.

Rgds,
Jean-Luc

[Jeff Schussele]

I neglected to also give credit for that Perf Cookbook paper to:

Doug Grumann, HP Performance Technology Center R&D lead engineer.

Stephen Ciullo is an HP Senior Technical Consultant & widely recognized as one of HP's best Performance experts.

Jeff

[Hein van den Heuvel]

My experiences, where absolutlely no measurable difference where found between a single versus 'a few' sections, come from
an SAP on Tru64 Unix environment.
Admittedly this is somewhat of a special case as SAP production sites have (very) long lasting DB connections.

Indeed if one were to frequently create slaves then having too many section will increase system time during process activation.

But, IMHO, adjectives like 'devastating' are a bit over the top to describe those effects and will set the wrong expectations.

I'm all for speeding up every single possible component in a system. But if attaching to the SGA has become a critical bottleneck in an application then I suggest that such application is overdue for a design review. Such application perhaps needs a transaction manager component or server slave tasks or whatever..

fwiw... on NUMA platforms, with a Numa aware oracle build, then whole picture changes as Oracle will create many more sections to fit in with the Numa environment.


Cheers,
Hein.

[Jeff Schussele]

Well if one would consider "devastating" a bit over the top, then one would have to consider terms like "baloney", "ghosts", "wild" & "insane" way over the top.

You really need to study PA-RISC architecture to understand how it handles memory mangement & process management structures & then further down into process structure layouts to have a good understanding as to just what impact - and the extent of such - would & will be caused in the scenario as I laid it out.

And as a side-note, this was an actual "real-world" scenario experienced by an actual HP customer. This was & is not theory. They had *serious* performance issues until the issue was solved by upping shmmax from 1GB to 10GB.

Rgds,
Jeff

[Todd McDaniel_1]

Stuart,

Having read your post again, I noticed that your have double the swap of your memory. If I can be so bold, I might suggest you can reclaim 50-75% of that space and opt for a bit more memory. Swap never needs to be larger than memory in my opinion and/or your /var/adm/crash FS.

My systems are configured for about 20% to 50% swap based on memory. The opposite ratio you have.

Regarding your question about shmmax, mine is configured at 1,073,741,824


My Superdome system is setup as such:

SWAP: 36GB
CPU Count: 48
CPU Speed: 750 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 73152 MB
bufpages: 2926 MB
maxuprc: 25320
maxvgs: 64
maxfiles: 2088
max_thread_proc: 256
nfile: 100010
nflock: 2448
nproc: 30000
ninode: 37332
shmmax: 1073741824
shmmni: 8192
dbc_max_pct: 4


MY N-class is setup as such:

SWAP: 5GB
CPU Count: 8
CPU Speed: 550 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 32768 MB
bufpages: 1310 MB
maxuprc: 4000
maxvgs: 255
maxfiles: 500
max_thread_proc: 256
nfile: 29539
nflock: 4096
nproc: 16020
ninode: 18468
shmmax: 1073741824
shmmni: 220
dbc_max_pct: 4

[Alzhy]

Todd, Is'nt it the maximum swap in an HPUX system is still at 32 GB?

Ours is set to be all 32 GB "regardless" of installed RAM since we are employing the 6-disk boot subsystem (mirrored root disks, mirrored exclusive swap disks, an ALTROOT disk and a hotspare disk).

And since we're using Cooked FIlesystems with DirectIO and RAW devices for ORacle Store ... our DBA's are maxing out SGA sizes .. so the reason for the 'enormous' SHMMAX declarations...

[Jeff Schussele]

Hi Nelson,

Absolute maximum swap in 11i is now 2TB.
That's if swchunk & maxswapchunks are at their max.
I have a system that I had to set swap at 128GB so all the applications could reserve when they start. They don't ever deactivate (swap) they just needed the reservation space. And yes they are rewriting it to not be so piggish when they start up.

Rgds,
Jeff

[Todd McDaniel_1]

I think for PRIMARY Swap it is 32GB max but you can have any amount of ALTERNATE swap, if I recall correctly.

Not sure what our DBAs use for the SGA size.

[Todd McDaniel_1]

hehe sorry forgot I had 11i yes it is 2TB...

I was mistakenly thinking of an older system I had.

NO points here...

[Hein van den Heuvel]

'over lunch' I made a reasonable (IMHO!) attempt to prove/disprove my assertion that it is good enough to have a reasonable shmmax. Thus this param could retain its 'fools guard' intent with low or no performance cost.

For now I see no significant difference when using a 4+GB SGA with shmmax set to 100mb, 1GB or 10GB. Admittedly my test has flaws...

What I did was to lauch N (set to 20) concurrent streams working their way through 100 equal, trivial commands (select sysdate, count(*) from v$session). This was done on an rx5670 4p 1.0 Ghz Itanium with 12GB physical memory running 11.23 and Oracle 9.2.0.2.0 - 64bit.

query: select to_char(sysdate,'HH24:MI:SS ') || count(*) as "x" from v$session;

SQL> startup
ORACLE instance started.
Total System Global Area 4874591504 bytes

The 'driver' process typically looked like:
time ./parallel z 20
real 15.5
user 2.9
sys 2.5

I also tried with 1, 5, 10 and 50 streams,
but this test serializes at 4 streams or so.

The initial setting happened to be 1GB-ish:
kctune -q shmmax
Tunable Value Expression Changes
shmmax 1073741824 Default Immed

ipcs -ma:
T ID CREATOR NATTCH SEGSZ
m 2088968 oracle92 8 1073741824
m 9 oracle92 8 1073741824
m 10 oracle92 8 1073741824
m 11 oracle92 8 1073741824
m 12 oracle92 8 663523328

During the run vmstat would look like:
faults cpu
in sy cs us sy id
1020 243 78 0 0 100
1162 9287 214 13 14 73
1290 15280 327 14 10 76
1409 20096 442 13 10 77
1487 24179 509 15 10 75
1557 27566 579 14 12 75


Next I switched to 10GB shmmax:

kctune shmmax=10737418240
Tunable Value Expression Changes
shmmax (before) 1073741824 Default Immed
(now) 10737418240 10737418240

ipcs -ma
T ID CREATOR NATTCH SEGSZ
m 2105352 oracle92 8 4891381760

vmstat:
faults cpu
sy cs us sy id
271 70 0 0 100
18495 287 14 13 73
22453 392 12 11 77
36510 769 15 11 74
37335 774 16 10 75
37978 793 15 11 74
38113 816 13 10 77


onwards to 100MB:
kctune shmmax=104857600
Tunable Value Expression Changes
shmmax (before) 10737418240 10737418240 Immed
(now) 104857600 104857600

ipcs -ma
T ID CREATOR CH SEGSZ CP
Shared Mem
m 2113544 oracle92 8 100663296
m 16393 oracle92 8 100663296
m 16394 oracle92 8 100663296
: <30+ lines removed>
m 65 oracle92 8 100663296
m 66 oracle92 8 9211904

vmstat
faults cpu
sy cs us sy id
272 70 0 0 100
269 67 7 9 84
15273 212 19 14 68
19671 325 14 10 76
24399 438 16 10 74
27095 524 14 10 76


Each time I took the second test run.
The first one, right after starting Oracle used more system time.

Flaw 1) I never saw more than 4 active sessions, where 20 where intended. The executed command took too little time. But... this _should_ amplify the startup time effects!

Flaw 2) The test never really 'touched' those 4GB. But.... when I added a large, in memory, sleect count(*) the user cpu time woudl totaly drawn out the system time.

Flaw 3) It's just a test! It is not representattive of any real situation, but I aimed for a worsed case.

Flaw 4) The anticipated increase in system calls (SHMGET) was not really visible in vmstat. If I feel like testing more some day I'll do a trace/truss/kaliper/whatever to understand better.


hth,
Hein.