OS failure message in Oracle

[John Jimenez]

Clay,
The Oracle admin found this. It looks like this might be the issues. I currently have MAX-ASYNC set to 200

Symptoms
The following messages in the alert log file and thereafter the instance crashed.

ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:socket failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: sskgxpcre1
Cause
This error means that there was an socket failure happened at the OS level while we try to communicate with
the remote node.

This issue can be caused if the MAX_ASYNC_PORTS has low value as it required to create a socket for a
useof the private interconnect when required.

MAX_ASYNC_PORTS specifies the system-wide maximum number of ports to the asynchronous disk I/O driver
that processes can have open at any given time. The default value for this parameter is 50.

http://docs.hp.com/en/939/KCParms/KCparam.MaxAsyncPorts.html
Solution
1. Set MAX_ASYNC_PORTS to a value high enough
2. Deactivate async io at os level.

[Volker Borowski]

Hi,

well you can start and stop the listener without having the database up and running, and the port ist set in listener.ora no matter what is in /etc/services.

And normaly you can start and stop the database with or without a listener being up and running.

If your listener died at 09:00 and your database died at 09:43:54, this suggests, somewhere is a memory leak.
So if you are lucky, the oracle patches may help. (btw from where to where do you want to patch?)

How is the dba trying to address the space you provide? Extending a tablespace with a new file ? in Filesystem or RAW ?

You have some alert.log stuff around that timestamp where the extension starts and where it "hangs" that you can attach?

Good hunting
Volker

[John Jimenez]

Thanks for the info Volker.

I feel all three issues we had were for different reasons. The database issues is resolved, but not we are having trouble keeping the listener up.
-Addressing the space? I do not know Oracle, but in the past the dba would create a new name FRA000 then FRA001 or DATA0000 then DATA001 then DATA002. But yesterday he tried to add it on to DATA0002. But because we had issues he will create a new DATA0003 instead.... but because of the issues will probably do this next week.
-Raw or file system? Everything having to do with RAC is using RAW, including the new LUNS. I change the permissions to one of the paths in /dev/rdsk and then the Oracle DBA sets it up.
-alert.log stuff? That just it I do not see any issues on the O.S. The error that I posted above was from the Oracle logs.

But I think this morning issue might had to do with MAX_ASYNC, because maybe there was garbage left over from yesterdays issue. I looked at some threads but have not been able to measure MAX_ASYNC. I opened up a new thread on MAX_ASYNC in the HP admin forum. I have not gotten an answer, so I just opened up work order with HP support too.

[John Jimenez]

shmmax is full on both systems.
shmmax 5368709120 / 5368709120

[Ariel Cary]

Hi John,

I think your reply on 'Nov 14, 2007 18:50:09 GMT' can lead to an answer.

Short answer: increase your swap space.

Longer one:
clearly, there is a lack of some OS resource(s):

ORA-27300: OS system dependent operation:socket failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: sskgxpcre1

when a process in HPUX is created or request more memory, the same amount of physical memory allocated is reserved in the swap area (such that it can be guaranteed to be used when the actual swapping happens); this is HPUX's behavior. If there is no swap space to reserve from, Oracle will fail to create processes, which could happen to user sessions, listener process, etc. or any other process that wishes to allocate more memory.

Output from swapinfo shows you have reserved all 8GB of total swap you have in your 16GB physical memory box. Thus, you must have at least 16GB of swap, even if you're not using it altogether.

HTH,

-Ariel

[Yogeeraj_1]

hi,

Waht version of Oracle are you running?

In oracle 10g, you have the Enterprise Manager Database control that provides will some online performance diagnostics as well as some more graphs on Host performance, Instance Disk IO and Instance Throughput.

You can get some indications there too.

e.g. Stats concerning the host performance:
Current Memory Page Scan Rate(pages/s) 0.3
Current Swap Utilization (%) 0.66

Paging Activity
Active Pages 74248522
Inactive Dirty Pages 51588930
Inactvie Clean Pages 0
Pages Paged-out (per second) 276.52
Pages Paged-in (per second) 5883.95
Pages Scanned by Page Stealing Daemon (per second) 21.28



please check
kind regards
yogeeraj

[John Jimenez]

Yes,

we have 10g. I will get with the Oracle Admin and see what he sees and I will post it.