Safe limits for shared memory allocation?

[Nick Battle_1]

We have a 32-bit application (on 11.00) that needs to use a large amount of shared memory. I need to estimate how large this shared segment can be before the system starts to fail.

I have the shminfo tool, and I understand the various MAGIC options for using quadrants 2,3 and 4, and memory windows - all of which may help me extend the limits.

What I'm having trouble finding is advice on how close to "full" I can run the system before normal operations - such as the memory mapping of shlibs by dld.sl - will fail. Obviously this will only fail when there isn't enough contiguous memory to satisfy the map in any quadrant, but how much is a safe amount to leave free to be reasonably sure of *not* running into trouble, yet not waste shared memory?

Any advice would be appreciated.

Cheers,
-nick

[Eric Antunes]

Hi Nick,

I think you are refering to kernel parameters shmmax, shmmni and shmseg? I use 2/3 of phisycal memory for shmmax (I don't know if you can use more but you can always try 3/4 and see...), 512 for shmmni and 120 for shmseg. Hope this will help!

Best Regards,

Eric Antunes

[Steven E. Protter]

You can safely max out shared memory allocation.

shmmax will never actually allocate beyond 25% of memory.

Memory is defined on HP-UX as ram plus swap.

You can do the calculation and set shmmax and see if its enough.

If not, add ram or swap and you can increase shared memory.

I've found on Oracle systems that ask for shmmax to be 100% of ram (ignorant of hpux restrictions I see) that on some very large systems I've worked with shared memory use never seemed to exceed 7%.

SEP

[Nick Battle_1]

Those kernel parameters let me tune how much memory is allocatable, but I need to know how close to 100% full I can go before I would expect programs to fail to execute - to map their libraries.

-nick

[Nick Battle_1]

I am told the following:

"Shared libraries are mapped first into quadrant 4, then into quadrant 3
if there is not enough contiguous space in Q4. If neither quadrant has
enough contiguous space then it will fail with errors like:

"/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/obam/lib/libobam.1
/usr/lib/dld.sl: Not enough space"

I don't want this to happen! But I don't know how close I can get to it happening safely. I think this is an issue about the 32-bit address space, not an issue regarding the amount of physical RAM or swap I have configured.

Am I missing something?

-nick

[Eric Antunes]

Are you running Oracle? Did you updated the Oracle software??

[Nick Battle_1]

Yes, we are running one Oracle instance. It is responsible for the majority of the allocation in quadrant 3.

-nick

[Eric Antunes]

But did you update any Oracle to an upper RSBMS version??

[Nick Battle_1]

The latest Oracle upgrade was to 9i. But I don't have a problem with Oracle itself. This is just a general question about how close you can safely run a system to its (32-bit) shared memory limits.

Let me explain again:

32-bit apps, collectively, can only allocate 1.75Gb of shared memory (if they're SHARED_MAGIC). Naively, if I write a 32-bit program that does a shmget for 1Gb, and then another shmget for 0.75Gb, it will work but there will no longer be *any* free shared memory *address space* left. (OK, it won't work because other things already use some of the 1.75Gb, but I'm simplifying here). When there is no shared address space left, no more 32-bit programs can run, assuming they use shared libraries that aren't already loaded. That's the dld.sl error I mentioned.

This would be a disaster!

So it's not safe to allocate the entire 1.75Gb, or anywhere near that. You need to allow some free space to let new programs run. Obviously how much you should leave depends on what programs you expect to run, but if the answer is "nothing unusual", what sort of safety-zone should you allow? 10Mb? 100Mb? 500Mb? Roughly. Ballpark.

The idea is to maximize the shm allocation for the program without threatening the stability of the system.

Cheers,
-nick

[Eric Antunes]

From HP-UX side SEP answered your question ("You can safely max out shared memory allocation."

From Oracle side, you must define shmmax large enough to hold entire SGA in just 1 shared memory segment. So what is the size of your SGA and your total physical memory?

Best Regards,

Eric Antunes

[Nick Battle_1]

> From HP-UX side SEP answered your question
> ("You can safely max out shared memory
> allocation.")

You're saying that I can allocate the entire 1.75Gb of 32-bit shared memory, then I can run another 32-bit program that wants to map new shared libs, and it will work just fine?

So what was that dld.sl error about then?

-nick

[Eric Antunes]

Not exactly:

From SEP, HP-UX will never allocate more than 0.25*(RAM+SWAP) for shared memory segments.

In my specific case I have 3,670,016 bytes of entire memory (RAM+SWAP) so I have only 917,504 for shared memory segments. I've defined a 256 Mb SGA but I could increase it to 512 Mb...

Best Regards,

Eric Antunes

[Don Morris_1 · Accepted solution]

Nick --

Your original interpretation is correct.
Processes operating in the (default) Global shared quadrants exhaust the map when creating shared objects, whether those objects are SysV Shared Memory, shared mmap (such as shared libraries), etc. If you eat up enough of the map (or fragment it) with shared memory, you can cause shared libraries to be unmappable.

Note that (as mentioned above) shared libraries default to Q4 first (because Q4 is the "most shared" 32-bit quadrant... unless you specify that you need it private, even Memory Windows users get the Global Q4), so exhausting Q4 is more serious than exhausting Q3.

Your best bet is for this application to run in a separate Memory Window from the system as a whole. Use SHMEM_MAGIC (I think that's the mnemonic for q2-shared) if you don't need more than 1Gb private space (text+data+stack+private mmaps), and that will give you 2Gb of window-specific shared space to work in without disrupting the Global Q4's 0.75Gb. If you can't share Q2 amongst your window, then you would want to stay below 1Gb to avoid risking system library issues in Q4.

Being more precise than that is impossible with the information given, I'm afraid. To know exactly how much you can allocate, we'd need to know exactly how large all the library mapping requests which follow would be, in what order they arrived, and exactly what addresses are free in the Global Window ranges (q3/q4) to account for fragmentation.

[Nick Battle_1]

I have 4Gb RAM and 12Gb swap, so I could have 4Gb shared (25%)... except I'm limited by the OS to 1.75Gb because of address space limitations, and my MAGIC model.

So I *could* ask for as much memory as HP-UX would every possibly allow. But *if* I do that, nothing else (32-bit) will be able to run. So I have to leave some unallocated.

How much? How can I go about finding out how much?

-nick

[Nick Battle_1]

Don,

Thanks very much for that. You've pretty much confirmed what I suspected.

One query though. You made it sound as though when Q4 was exhausted, shlibs wouldn't be mapped from Q3 - is that right?

Memory windows seem to be the way to go medium term (64-bit is long term). Before then, it looks like the app will fill Q4 and not be able to use Q3 (it needs contiguous space). That leaves quite a lot in Q3 that would save us - assuming shlibs *can* be allocated from Q3. So I may never have to work out the size of the minimum safe zone, as I can't use 100% of the memory without redesigning the app.

Cheers,
-nick

[Nick Battle_1]

Thanks everyone.

Basically, the minimum safe limit really just depends on what I want to run. Because of the app's design and the layout of Q3/Q4, in practice I will have ~130Mb in Q3 that I can never use (without changing the app). The total shared library portion from Q4 is about 64Mb, so I'm relatively happy to assume that the 130Mb will be sufficient for "normal use".

Cheers,
-nick