1748195 Members
4578 Online
108759 Solutions
New Discussion юеВ

Re: error in alert log.

 
SOLVED
Go to solution
AnthonySN
Respected Contributor

error in alert log.

we have a 2 node cluster with OS hpux 11.31,mc/sg and oracle 10g
we are getting the below errors in the alert log and the application needs to be restarted because it hangs.

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed


tail -20 alert_2K.log
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:47 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:47 2010
ksvcreate: Process(q001) creation failed
Tue Feb 16 11:07:58 2010
Process startup failed, error stack:
Tue Feb 16 11:07:58 2010
Errors in file ...._psp0_22791.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed
26 REPLIES 26
TTr
Honored Contributor

Re: error in alert log.

It looks like you have hit a system limit and can your oracle account can not start up anymore processes.

Check you /var/adm/syslog/syslog.com for more information. You may have to increase some kernel parameters such as nproc, maxuprcs, nfile, nflocks etc. Also check you memory and swap space utilization.
Dennis Handly
Acclaimed Contributor

Re: error in alert log.

>TTr: Also check your memory and swap space utilization.

Right, you want to do a "swapinfo -tam" at the same time as your heavy load, to be as accurate as possible.
Kapil Jha
Honored Contributor

Re: error in alert log.

Is it a new setup and if yes have u tunned various kernel parameters in the server.

Plus check your memory usage when you trying to start the oracle.

BR,
Kapil+
I am in this small bowl, I wane see the real world......
Horia Chirculescu
Honored Contributor

Re: error in alert log.

Hello, Sasj,

Please read the following document related to recommended kernel parameters:

http://h21007.www2.hp.com/portal/download/files/unprot/database/HP3KOracle.ppt

Make the settings and then try to start again your instance. If still have problems, should check also oracle initialization parameters

Best regards,
Horia.
Best regards from Romania,
Horia.
AnthonySN
Respected Contributor

Re: error in alert log.

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 380 16004 2% 0 - 1 /dev/vg00/lvol2
reserve - 16002 -16002
memory 15555 15247 308 98%
total 31939 31629 310 99% - 0 -

We have noticed that there is no swapping even when the memory reaches 100% utilization.
also changed nproc value. and have reduced the filecache_max/min to 5 & 3 % respectively.
Steven Schweda
Honored Contributor

Re: error in alert log.

> [...] fork failed with status: 11

Around here, error 11 is EAGAIN, "No more
processes". I'd worry less about memory,
and more about a limit on process count. Of
course, the error code could be misleading.

> [...] also changed nproc value.

From what to what?

man ulimit
AnthonySN
Respected Contributor

Re: error in alert log.

from 4200 to 5000
Horia Chirculescu
Honored Contributor

Re: error in alert log.

Can you check for Bug 3411021 Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 ?

It seems that this bug is fixed on 10.2.0.3+.

What is your version of oracle?

Horia.
Best regards from Romania,
Horia.
AnthonySN
Respected Contributor

Re: error in alert log.

oracle 10gR2.

the problem occurs only when the swapinfo shows 100% utilization of memory.
ram and swap is 16GB each.
why is there no swapping.