topic Re: shmmax with Oracle Applications on hp-ux 11.11. in Operating System - HP-UX

shmmax with Oracle Applications on hp-ux 11.11.

Stuart Abramson_2 — Thu, 20 Nov 2003 08:24:37 GMT

I just read a ticket on shmmax with 32-bit oracle applications.

What should I set shmmax to if I'm running full 64-bit Oracle Applications 11i with Oracle 9i on hp-ux 11.11 with 12 GB RAM and 24 GB swap?

Someone suggested full memory + swap. In my case that would be 36 GB!

Re: shmmax with Oracle Applications on hp-ux 11.11.

Tomek Gryszkiewicz — Thu, 20 Nov 2003 08:37:39 GMT

On my Oracle box (20 procs, 40GB) we set it to about 17GB - this value was suggested by one of the "Oracloids" :-)

-Tomek

Re: shmmax with Oracle Applications on hp-ux 11.11.

Alzhy — Thu, 20 Nov 2003 08:38:48 GMT

That would be correct. Although I doubt any DBA would actually size an Oracle instance to have an SGA (which takes 1 shared memory segment chunk)that will exceed available RAM.

For 32-Bit applications, it really does not matter but most Oracle documentation would recommend setting it to 4 GB (max address space for 32-bit apps?) regardless of memory installed .. Of course rarely these dyas you will find a DB server whose total VM is less than 4 GB anyway.

If you will be having multiple DB instances, make sure the total SGA sizes will not exceed installed RAM or encroach into swap area(s) -- use "swapinfo -atm". If you see your swap devices being used already -- chances are your total sum of your SHMEM segments (SGA mostly) have exceeded RAM and you're most likely be memory bound already.

Re: shmmax with Oracle Applications on hp-ux 11.11.

Stuart Abramson_2 — Thu, 20 Nov 2003 08:39:28 GMT

Did they have some kind of formula?

Re: shmmax with Oracle Applications on hp-ux 11.11.

Stuart Abramson_2 — Thu, 20 Nov 2003 08:43:57 GMT

Nelson, you said: "Although I doubt any DBA would actually size an Oracle instance to have an SGA (which takes 1 shared memory segment chunk)that will exceed available RAM."

Then:
1. If my RAM in 12 GB, but I use some of it for kernel space, process space, and buffer cache, I want to set my "shmmax" to, let's say, 8 GB.

2. If I have two (equal size) SIDs on the same machine, I would then set "shmmax" to 4 GB.

Does that sound right?

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 08:46:19 GMT

Hi Stuart,

I would find out what size SGA the DBAs are using & that would be the absolute minimum.
Depending on what that size is, throw extra in to cover any non-Oracle needs.

I've just heard from very reliable sources that Oracle 9i has some new tricks up it's sleeve that can really cause problems if you're not paying close attention
For EX:
Let's say the DBAs have set up a 10GB SGA & shmmax is set to the default 1GB. Oracle 9i will now - instead of erroring out - go ahead & create 10 1GB shmem segments & the resultant performance hit is devastating.

So consult with the DBAs on this & throw in extra for other overhead.

HTH,
Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Alzhy — Thu, 20 Nov 2003 08:48:26 GMT

Stuart,
Bear in mind that SHMMAX is not immediately allocated on boot time. It is just a guidance as to how large of a Shared Memory Segment can the kernel allow to be created. You can even declare it to be well larger than your VM (swap + memory) and the kernel will not complain. If I were you, the safest will be to set it to the size of your RAM. FOr some breathing room -- you could add half of your swap -- so 24 GB total SHMMAX.

Again, SHMMAX is not allocated at boot time. It is just a guidance.

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jean-Luc Oudart — Thu, 20 Nov 2003 08:57:52 GMT

As mentioned, SHMMAX is a high water mark and the SGA should be limited by this value.

on 9i, the recommended size is same as available memory on the server.

source =metalink doc id 169706.1
Rgds,
JL

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 09:22:00 GMT

Hi Nelson/JL,

I don't agree that it should be set to those high values. I view shmmax the same as the maxXsiz parms - they're fences - fences to keep processes/application from monopolizing or exhausting system resources. In this case RAM. And they should be set to reasonable values agreed to by all parties & then all must be required to adhere to them.
Applications are like buffer cache in a sense, they're going to use all you allow them to & if you give them too much - they'll use it & the resultant consequences may not be desirable.

My 2 cents,
Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Hein van den Heuvel — Thu, 20 Nov 2003 09:35:05 GMT

Jeff wrote: "Let's say the DBAs have set up a 10GB SGA & shmmax is set to the default 1GB. Oracle 9i will now - instead of erroring out - go ahead & create 10 1GB shmem segments & the resultant performance hit is devastating."

Other replies imply this.

Balloney. You guys are seeing ghosts.
I challenge you to actually measure the difference between 1*10GB and 10*1GB.
Now if you made that 1000*10mb then that woudl matter, but as long as you have picked a reasonable value ( 500MB - 4GB ) then you will not be able to tell teh difference. Just let Oracle create multiple segments. No harm done and be more assured that an other application will not go too wild.

The suggestion to set shmmax to physical memory + swap borders on the insane!

hth,
Hein.

Re: shmmax with Oracle Applications on hp-ux 11.11.

Alzhy — Thu, 20 Nov 2003 09:46:46 GMT

Agreed Hein, I actually never have seen any actual metrics as to how Oracle performs when SGA is in just 1 SHM segment or some "saner" number of segments. The suggestion to set SHMMAX to as large as VM is more largely a safety net for malformed and misbehaving applications (and instances?)so we as admins can analyze effectively during such situations. My rule of thumb is to always ask the DBA's what biggest SGA is expected on a system and set it to that value. We also set a minimum of 1. But I guess with 11i with cooked Filesystems - the norm these days is to always have the biggest SGA that can possiby be allocated for an instance. I leave it out the the HPUX doctorates and laureates to have a more exhaustive assessment of which is correct.

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 10:16:35 GMT

Hi Hein,

With all due respect I believe you're wrong.
Do you have any idea the costs associated with the shmset(), shmget() & shmat() system calls? Simply imagine a process needs to attach to 10 shmem segments & another needs to only attach to one. Which one do you think will be faster? Then the overhead associated with working down through process structures that are attached to 10 separate shared memory segments as opposed to only one. Again which one will be doing more work in your opinion? And remember this will all be System CPU time which will be stolen from the User CPU time.

I really think you'd benefit from attending one of the great HP classes like HP-UX Internals or even the Performance & Tuning Class.

And here's a great paper from Stephen Ciullo that is very informative:

http://h21007.www2.hp.com/dspp/ddl/ddl_Download_File_TRX/1,1249,280,00.pdf

Rgds,
Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jean-Luc Oudart — Thu, 20 Nov 2003 10:32:20 GMT

Jeff,

as I said this a high water mark only.

The right process should be proper planning and communication between Sysadmin and DBA.

Rgds,
Jean-Luc

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 10:45:55 GMT

I neglected to also give credit for that Perf Cookbook paper to:

Doug Grumann, HP Performance Technology Center R&D lead engineer.

Stephen Ciullo is an HP Senior Technical Consultant & widely recognized as one of HP's best Performance experts.

Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Hein van den Heuvel — Thu, 20 Nov 2003 10:56:07 GMT

My experiences, where absolutlely no measurable difference where found between a single versus 'a few' sections, come from
an SAP on Tru64 Unix environment.
Admittedly this is somewhat of a special case as SAP production sites have (very) long lasting DB connections.

Indeed if one were to frequently create slaves then having too many section will increase system time during process activation.

But, IMHO, adjectives like 'devastating' are a bit over the top to describe those effects and will set the wrong expectations.

I'm all for speeding up every single possible component in a system. But if attaching to the SGA has become a critical bottleneck in an application then I suggest that such application is overdue for a design review. Such application perhaps needs a transaction manager component or server slave tasks or whatever..

fwiw... on NUMA platforms, with a Numa aware oracle build, then whole picture changes as Oracle will create many more sections to fit in with the Numa environment.

Cheers,
Hein.

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 11:34:18 GMT

Well if one would consider "devastating" a bit over the top, then one would have to consider terms like "baloney", "ghosts", "wild" & "insane" way over the top.

You really need to study PA-RISC architecture to understand how it handles memory mangement & process management structures & then further down into process structure layouts to have a good understanding as to just what impact - and the extent of such - would & will be caused in the scenario as I laid it out.

And as a side-note, this was an actual "real-world" scenario experienced by an actual HP customer. This was & is not theory. They had *serious* performance issues until the issue was solved by upping shmmax from 1GB to 10GB.

Rgds,
Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Todd McDaniel_1 — Thu, 20 Nov 2003 12:04:30 GMT

Stuart,

Having read your post again, I noticed that your have double the swap of your memory. If I can be so bold, I might suggest you can reclaim 50-75% of that space and opt for a bit more memory. Swap never needs to be larger than memory in my opinion and/or your /var/adm/crash FS.

My systems are configured for about 20% to 50% swap based on memory. The opposite ratio you have.

Regarding your question about shmmax, mine is configured at 1,073,741,824

My Superdome system is setup as such:

SWAP: 36GB
CPU Count: 48
CPU Speed: 750 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 73152 MB
bufpages: 2926 MB
maxuprc: 25320
maxvgs: 64
maxfiles: 2088
max_thread_proc: 256
nfile: 100010
nflock: 2448
nproc: 30000
ninode: 37332
shmmax: 1073741824
shmmni: 8192
dbc_max_pct: 4

MY N-class is setup as such:

SWAP: 5GB
CPU Count: 8
CPU Speed: 550 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 32768 MB
bufpages: 1310 MB
maxuprc: 4000
maxvgs: 255
maxfiles: 500
max_thread_proc: 256
nfile: 29539
nflock: 4096
nproc: 16020
ninode: 18468
shmmax: 1073741824
shmmni: 220
dbc_max_pct: 4

Re: shmmax with Oracle Applications on hp-ux 11.11.

Alzhy — Thu, 20 Nov 2003 12:11:02 GMT

Todd, Is'nt it the maximum swap in an HPUX system is still at 32 GB?

Ours is set to be all 32 GB "regardless" of installed RAM since we are employing the 6-disk boot subsystem (mirrored root disks, mirrored exclusive swap disks, an ALTROOT disk and a hotspare disk).

And since we're using Cooked FIlesystems with DirectIO and RAW devices for ORacle Store ... our DBA's are maxing out SGA sizes .. so the reason for the 'enormous' SHMMAX declarations...

Re: shmmax with Oracle Applications on hp-ux 11.11.

Jeff Schussele — Thu, 20 Nov 2003 12:21:17 GMT

Hi Nelson,

Absolute maximum swap in 11i is now 2TB.
That's if swchunk & maxswapchunks are at their max.
I have a system that I had to set swap at 128GB so all the applications could reserve when they start. They don't ever deactivate (swap) they just needed the reservation space. And yes they are rewriting it to not be so piggish when they start up.

Rgds,
Jeff

Re: shmmax with Oracle Applications on hp-ux 11.11.

Todd McDaniel_1 — Thu, 20 Nov 2003 12:26:27 GMT

I think for PRIMARY Swap it is 32GB max but you can have any amount of ALTERNATE swap, if I recall correctly.

Not sure what our DBAs use for the SGA size.