Strange error on HP-UX with Oracle

[Maarten van Maanen]

This morning we had a strange error when we tried to startup Oracle. The error is listed below.

SVRMGR> startup
ORA-07279: spcre: semget error, unable to get first semaphore set.
HP-UX Error: 28: No space left on device

All volumes on our HP-UX machine had enough space left (as usual) to run Oracle. Our next idea was to check the semaphores, but according to glance, only 23 out of a possible 600 configured in the kernel were used.
After a complete reboot of the system Oracle could be started with no problems. So, things are working again but I still don't know what the exact problem was so any help on this is welcome.

[Rob Smith]

I have seen this error before but usually it means that the system has run out of semaphores. There is a known problem with Oracle where it does not release semaphores after stopping/starting the app. Did you look at the output of ipcs to see how many semaphores were claimed by Oracle? When was the last time you bounced the box? Last time I talked to somebody at Oracle they told me the solution was to restart the server more often, I was a little disappointed at that response.

[nancy rippey]

I have seen these error before and was able to correct it by increasing the semmns and semmni
kernel parameters.

[Maarten van Maanen]

Uptime of the HP-UX was 99 days (I've Novell Netware servers up and running almost 200 days with no problems). An answer like 'just reboot more often' is not very acceptable to me.

I did not check ipsc but looked in Glance instead. Glance told me only 22 semaphores were in use.

[Rita C Workman]

There can be any number of reasons why those semaphores are not releasing properly...here a couple.
1. You may need to check to ensure your patches are current and configured....for HPUX and Oracle.
2. Your users may not be shutting the application properly - just blindly closing out windows with application running instead of properly logging out.
3. The actual application may be poorly coded so as to not release semaphores properly.
...an so on...
The obvious option, rebooting, you have tried. What you may need to do is start checking for the above conditions and see what you come up with. In the meantime you can release things using the ipcrm command as a start to avoid rebooting.
....but generally it's the time it takes to track down the cause that buys the best results. Many times it's just end users ...

Hope this helps,
/rcw

[nancy rippey]

In my case when experiencing the problem glance also stated that the system was not out of semaphores. This occured on more than one server. Oracle and HP's recommendation was to up the number of semmaphores,(semmni and semmnu). If running Tuxedo the semaphones will need to be further increased. On our servers running Oracle 7.3.4 and Oracle 8.1.7 the semaphore parameters are as follows:
semmni 4980
semmns 15000
semmnu 2000
For servers running Oracle and Tuxedo the
semaphones may need to increased even further.

Hope this helps

[Steven Sim Kok Leong]

Hi,

Please check your shared memory as well via ipcs -b -m. If during the last Oracle shutdown the shared memory was not released, then you will not be able to startup Oracle properly.

If your shared memory is in use by Oracle (not detached with a D and present in ipcs -b -m) even when your Oracle is not started up, perform ipcrm -M shmid (ie. shared memory id).

Subsequently, try starting up your Oracle via svrmgrl.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Stephen Bouzan]

Just a side note on server uptime: Running a server 99 days without a reboot is unwise. We reboot our production N class servers once a week running (they run Oracle). That way we know that our servers can boot and that the boot files are not corrupt. I mean if you think about it, how would you tell you boss that you haven't known for the past 200 days if you server has had the ability to boot. This is a good Sys Admin practice.

[Wodisch]

Hello Maarten,

check how many semaphores and how much shared
memory is occupied before you even try to start Oracle:
ipcs -sb -mb

If you only have one instance, there should be nothing
being owned by "oracle" (or whatever your local admin
user for Oracle is). IF there some, try to remove them
with "ipcrm" or a "shutdown abort". If you cannot, it is
time for a reboot, really (but not before).

If no semaphores, etc, were allocated, check your local
init*ora file: has somebody increased "processes"?
That is the 1:1 parameter to semaphores needed by
the instance. If it is bigger (or even only almost as big
as) than your kernels "semmns", then you have to build
a new kernel with a bigger value for "semmns" (and then
you have to reboot to install the kernel).

HTH,
Wodisch