OS failure message in Oracle

[John Jimenez]

Can anyone interpit what this means? It is a Memory issue that came from Oracle. I cannot find any issues in any of my HPUX 11.11 logs. but Oracle says it is an OS issue so I am trying to resolve

Errors in file /opt/oracle/admin/MXPROD/udump/mxprod1_ora_655.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27544: Failed to map memory region for export
ORA-27300: OS system dependent operation:socket failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: sskgxpcre1
Wed Nov 14 09:43:54 2007

[Michael Mike Reaser]

The cynic in me wants to point out that Oracle will **ALWAYS** claim it's an OS issue. Of course, can they tell you *what* that "OS issue" might be?

No. All that matters is that they have now proclaimed that it's **NOT** an "Oracle issue".

[Murat SULUHAN]

Hi John

ORA-00603: ORACLE server session terminated by fatal error

ORA-00603 problem is Internal Oracle error so you should consult to Oracle support. They may give you specific or general patch

Best Regards
Murat

[A. Clay Stephenson]

If the error number is 11 (EAGAIN) then that suggests that the system-wide number of processes limit (NPROC) or the per-user number of processes (MAXUPRC) has been reached. Examine those tunables and see if they need increasing. (I'm leaning towards maxuprc.)

The bad news is that '11' could also be a signal number and signal 11 is SIGSEGV is a segmentation violation which means that a process was trying to use memory it wasn't supposed to --- which almost allways means bad code --- which could be Oracle itself or a shared library.

However, because I see 'Resource temporaily unavailable' that strongs suggests that this '11' is indeed EAGAIN. Running the process under tusc would allow you to see exactly which system call is failing and thus know the tunable to set.

[John Jimenez]

He is having listener issues

Here is the long story. Last week I created 2 50 gig LUNS on the SAN for the Oracle Admin to use for (we have 2 RP7420 using RAC). He ran into some issues adding this on to data002 it just hung. In the past I have noticed that the usually created a new data or fra when I gave them disk space, not add them to a current one. So I think it was a Oracle procedural reason for the failure. He canceled it and it created a zombie on the O.S. An hour later everyone locked up. We rebooted the Server with the zombie, but ran into a 2nd issues 2 months when I created 2 other 50 gig luns which they used for data2, they did not put these 2 groups in the start up script, so it did not mount. He fixed it, but to make sure we rebooted both servers one at a time last night. This morning at 9:00 the listner went down and people were not able to connect again. One thing that has always confused me is that the listener uses port 1521, but I never had this in /etc/services file

[John Jimenez]

Current info from Glance. One thing that is weird on this system is that everone comes in from the application that is on Windows as the same Login, if I do a "who I have one entry for MAXUX but there is actually 150 people using it.
Here is the glance output, nothing seems bottlenect system wide, but maybe since it thinks there is only one user some other kernal is maxed out.

SYSTEM TABLES REPORT Users= 5

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4200 736 18 18
File Table (nfile) 65536 7077 11 11
Shared Mem Table (shmmni) 512 10 2 2
Message Table (msgmni) 4200 2 0 0
Semaphore Table (semmni) 4096 25 1 1
File Locks (nflocks) 4200 7 0 0
Pseudo Terminals (npty) 60 0 0 0
Buffer Headers (nbuf) na 215109 na na

[John Jimenez]

Just to correct the "long story" Last week I created the LUN, Yesterday is when the oracle admin trying to add it and we started having issues.

[John Jimenez]

Murat, Funny you should mention patches. Last month my O.S. Patches were 12 months behind, but are now caught up. The patches on Oracle are way behind and are going to be scheduled soon.

[John Jimenez]

FYI swapinfo

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 4 8188 0% 0 - 1 /dev/vg00/lvol2
reserve - 8188 -8188
memory 16357 13741 2616 84%
total 24549 21933 2616 89% - 0 -

[John Jimenez]

Another correction...sorry the 2 servers are on 11.23 not 11.11.
Clay, I think you were on to something on the NPROC, because everone comes in with user MAXMC, NPROC = 4200 and maxuprc = 3700 but glance shows this to only be at 18%. If you have any other idea's can you let me know? thank you