Getting memory errors after adding two more Oracle instances.

[A. Clay Stephenson]

Important question. Are you running 32-bit Oracle or applications? I suspect that your memory is too fragmented to allow for an additional large contigious memory allocations.

[Patrick Wallek]

Some more info please....
1) What type of system?
2) What version of HP-UX?
3) 32 or 64 bit?
4) What version of Oracle? Is it 32 or 64 bit?

My first reaction is that you need more memory. 4GB does not seem like very much for 10 Oracle instances running.

Your swap is not being used at all. You are fine there.

I have seen a problem similar to this, but with the Unidata database system giving messages "No more core". The problem was that we were running 32-bit Unidata. 32-bit applications can only access a maximum of 2.7GB of shared memory.

If you are running 32-bit Oracle, I would bet that this is your problem. You are hitting the maximum shared memory that 32-bit applications can use. When you start another instance and it tries to access more shared memory, it is not available and you are getting the error.

[James R. Ferguson]

Hi:

Well Clay and Patrick have already convered he what and the why I suspect.

As 'maxdsiz' this applies to 32-bit processes. Its maximum is 0x7B03A000 (~ 2 GB). 'maxdsiz_64bit' applies (of course) to 64-bit processes.

Regards!

...JRF...

[Jack C. Mahaffey]

Thanks for the feedback.

The server is HP-UX 11.00 64-bit running
Oracle 7.3.4. It is a development-only system. SGA's for the databases are around 60mb each. Not very large.

The problem is showing up with the applications. syslogd even quit yesterday when we were having the problems. No errors in syslog or alert log.

My first recommendation will be to up the maxdsize to 2gb to match the maxdsize_64-bit value. I will also recommend changing the timeslice from 10 to 1.

I will likely have a window next week to rebuild the kernel.


I suspect Oracle and the applications are 32-bit. CD's for both previously ran on 10.20. How do I tell?

Thanks again... Jack

[A. Clay Stephenson · Accepted solution]

Yep, your problem is 32-bit memory limits. I suppose that you want to set timeslice from 10 to 1 because your box is running much too fast and you would like to kill its performance. Leave it at 10 or you are going to do context switching and very little else.

Regards, Clay

[John Palmer]

Hi,

Oracle 7.3.4 is 32bit only. I suspect that you are hitting the system-wide 32bit shared memory limit of 1.75Gb. You can check what shared memory segments exist with the command 'ipcs -am'.

If you can't reduce the size of your Oracle SGA's, you'll have to investigate the HP-UX 11 'memory windows' facility which allows you to have several different shared memory 'windows' all up to 1.75Gb. If you search for 'shared memory' and 'memory windows' you'll find lots of useful information.

Regards,
John

[Jack C. Mahaffey]

One more question, is the limit of 1.75Gb a limit for all 32-bit applications regardless if it related to a database or not?

jack...

[John Palmer]

Yes, the total of the SEGSZ column is the total shared memory in use. As you can see, you've got a lot of segments allocated in the default 1.75GB. With this many segments, shared memory also gets fragmented such that even though a large amount may be available, it's split into lots of smaller chunks.

Oracle does try to allocate a number of smaller segments if a single large segment isn't available although this may have been implemented since 7.3.4.

Your system will definitely benefit from having multiple memory windows although you'll have to write the appropriate control scripts.

Regards,
John

[A. Clay Stephenson]

That's the limit for all 32-bit processes on the system. Now what you can do under the 64-bit OS is set up "Memory Windows" so that each process (or group of related 32-bit processes) get it's own 32-bit address space; no one process can exceed the normal 32-bit restrictions but your are then freed from the system-wide limits that you are now operating under. Note that you may not even get nearly the address space that your are requesting via shmget() because there is simply not that big a 'slice' available eventhough the sum of the availables 'slices' exceeds your shmget() request. Memory windows will definitely help you. Search on "memory windows" and you should find everything you need.

[A. Clay Stephenson]

That's the limit for all 32-bit processes on the system. Now what you can do under the 64-bit OS is set up "Memory Windows" so that each process (or group of related 32-bit processes) get it's own 32-bit address space; no one process can exceed the normal 32-bit restrictions but you are then freed from the system-wide limits that you are now operating under. Note that you may not even get nearly the address space that your are requesting via shmget() because there is simply not that big a 'slice' available eventhough the sum of the availables 'slices' exceeds your shmget() request. Memory windows will definitely help you. Search on "memory windows" and you should find everything you need.

[John Palmer]

To answer your last question... yes the 1.75Gb limit applies to all 32 bit processes. Databases (like Oracle) just make use of shared memory (usually lots of it).

Regards,
John

[Patrick Wallek]

The shared memory limit is the total that is allocated for ALL 32-bit applications.

It is NOT 1.75 GB for EACH application. It's for ALL. That is the limit for everything.

I would follow John's suggestion and look into memory windows.

[Jack C. Mahaffey]

Thanks all for the great help. I'll need to look into memory windows, how to configure them, etc.



Thanks again...

[Bill Hassell]

Oracle 7.xx is really old and yes, it is 32bit only which severely constrains what memory can be used by the applications. Since I see mqm as an owner, I suspect that MQM is also grabbing shared memory from the one single shared memory area--there is no way to add any more memory to this window since it is constrained by 32bit addressing limits.

This is going to take a lot of work to resolve. Each instance of Oracle will have to be started in in a separate memory window--which is good because no one else competes for that space (like memory mapped files, shared libraries, other shared memory segments), but you (and the DBA) will have to sort out everything that attaches to this particular instance of Oracle. The reason is that these windows are known only to the processes that start inside them--the rest of the 32bit world see memory window #0, the default window.

Here is the tool that ipcs cannot provide:

ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

With this program, you'll see exactly how fragmented the window is and all the tiny components that prevent allocating large chunks (which must all be contiguous)

All of these hoops are due to 32bits...upgrade to Oracle 32bit (8i or beyond) and all of these issues disappear. Of course, all supporting apps will have to be converted but since 64bit HP-UX has been around since Nov 1997, all those apps have long since been converted. (or I would certainly hope so)