Simpler Navigation for Servers and Operating Systems
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
cancel
Showing results for 
Search instead for 
Did you mean: 

error in alert log.

SOLVED
Go to solution
AnthonySN
Respected Contributor

error in alert log.

we have a 2 node cluster with OS hpux 11.31,mc/sg and oracle 10g
we are getting the below errors in the alert log and the application needs to be restarted because it hangs.

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed


tail -20 alert_2K.log
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:47 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:47 2010
ksvcreate: Process(q001) creation failed
Tue Feb 16 11:07:58 2010
Process startup failed, error stack:
Tue Feb 16 11:07:58 2010
Errors in file ...._psp0_22791.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed
26 REPLIES
TTr
Honored Contributor

Re: error in alert log.

It looks like you have hit a system limit and can your oracle account can not start up anymore processes.

Check you /var/adm/syslog/syslog.com for more information. You may have to increase some kernel parameters such as nproc, maxuprcs, nfile, nflocks etc. Also check you memory and swap space utilization.
Dennis Handly
Acclaimed Contributor

Re: error in alert log.

>TTr: Also check your memory and swap space utilization.

Right, you want to do a "swapinfo -tam" at the same time as your heavy load, to be as accurate as possible.
Kapil Jha
Honored Contributor

Re: error in alert log.

Is it a new setup and if yes have u tunned various kernel parameters in the server.

Plus check your memory usage when you trying to start the oracle.

BR,
Kapil+
I am in this small bowl, I wane see the real world......
Horia Chirculescu
Honored Contributor

Re: error in alert log.

Hello, Sasj,

Please read the following document related to recommended kernel parameters:

http://h21007.www2.hp.com/portal/download/files/unprot/database/HP3KOracle.ppt

Make the settings and then try to start again your instance. If still have problems, should check also oracle initialization parameters

Best regards,
Horia.
Best regards from Romania,
Horia.
AnthonySN
Respected Contributor

Re: error in alert log.

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 380 16004 2% 0 - 1 /dev/vg00/lvol2
reserve - 16002 -16002
memory 15555 15247 308 98%
total 31939 31629 310 99% - 0 -

We have noticed that there is no swapping even when the memory reaches 100% utilization.
also changed nproc value. and have reduced the filecache_max/min to 5 & 3 % respectively.
Steven Schweda
Honored Contributor

Re: error in alert log.

> [...] fork failed with status: 11

Around here, error 11 is EAGAIN, "No more
processes". I'd worry less about memory,
and more about a limit on process count. Of
course, the error code could be misleading.

> [...] also changed nproc value.

From what to what?

man ulimit
AnthonySN
Respected Contributor

Re: error in alert log.

from 4200 to 5000
Horia Chirculescu
Honored Contributor

Re: error in alert log.

Can you check for Bug 3411021 Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 ?

It seems that this bug is fixed on 10.2.0.3+.

What is your version of oracle?

Horia.
Best regards from Romania,
Horia.
AnthonySN
Respected Contributor

Re: error in alert log.

oracle 10gR2.

the problem occurs only when the swapinfo shows 100% utilization of memory.
ram and swap is 16GB each.
why is there no swapping.
TTr
Honored Contributor

Re: error in alert log.

> reserve - 16002 -16002

It looks like your address space may have been exhausted.
There may not be active disk swapping which means that you will not have i/o issues but almost all your swap space is reserved by the running processes.

You need to add more swap space.

After you add more swap space, more processes will run which means that the server most likely will swap to disk actively and the performance will decrease tremendously. So adding more memory would be a good idea.
AnthonySN
Respected Contributor

Re: error in alert log.

TTr> but almost all your swap space is reserved by the running processes.

which processes you mean, which is that process which requires 16GB swap reservation.
TTr
Honored Contributor

Re: error in alert log.

> which processes you mean, which is that process which requires 16GB swap reservation

All of the running processes combined.

In the meantime you can look in the running processes and see if you can clean up any of them or turn off any unnecessary services.
AnthonySN
Respected Contributor

Re: error in alert log.

the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram it would be not justifiable to ask for more physical memory.
AnthonySN
Respected Contributor

Re: error in alert log.

TTr>>
> reserve - 16002 -16002
It looks like your address space may have been exhausted.

may you pls shed some light on the above.
is there any way to reduce the reserve.
how can the d/b start using swap space if short of memory. and what is address space ?
Dennis Handly
Acclaimed Contributor
Solution

Re: error in alert log.

>is there any way to reduce the reserve?

Only by running fewer processes, or smaller/fewer shared memory segments.

>how can the db start using swap space if short of memory.

You have to have more unreserved device swap.

>and what is address space?

In this case, I assume TTr really meant reserved device swap used up all of the device swap.
AnthonySN
Respected Contributor

Re: error in alert log.

> Only by running fewer processes, or smaller/fewer shared memory segments.

how can we reduce those processes?
we have checked the services running and they are bare minimum, its a cluster and ora10g running on it, whenever the mumber of sessions reaches 1600+ the memory utilization touches 100%.
and why is that swap utilization shows only 3% used.
we need strong reasons to justify the additional memory needed.
Dennis Handly
Acclaimed Contributor

Re: error in alert log.

>the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram

What does "swapinfo -tam" show for both machines?
Did you configure your Oracle SGA too large on your new machine?
Dennis Handly
Acclaimed Contributor

Re: error in alert log.

>how can we reduce those processes?

By killing them or not starting them. Or setting some configuration value to not do so many.

>number of sessions reaches 1600+ the memory utilization touches 100%.

Less sessions?

>why is that swap utilization shows only 3% used.

You have reserved that space for growth. If you don't want to grow, you shouldn't reserve so much.

>we need strong reasons to justify the additional memory needed.

You can just add more device swap first and check your performance.
AnthonySN
Respected Contributor

Re: error in alert log.

the application is now migrated to this new system so i cant get the swapinfo -tam output now.

sga is 2.5 gb
AnthonySN
Respected Contributor

Re: error in alert log.

>Or setting some configuration value to not do so many.
have changed nproc, what else do you suggest.

>Less sessions?
cant be done since this application is used by all over the branches.

>You have reserved that space for growth. If you don't want to grow, you shouldn't reserve so much.
How do i unreserve.
Dennis Handly
Acclaimed Contributor

Re: error in alert log.

>so I cant get the swapinfo -tam output now.

Is your SGA value larger now?

You may want to look at these recent threads and see how to use ps(1) to see how much VM is being used by each process:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1411399
TTr
Honored Contributor

Re: error in alert log.

> the application is now migrated to this new system

Did you also upgrade the oracle version? And/or the application version?

With the information given so far, there is nothing wrong with your setup. Either your application environment has grown (much more data, larger database, more users, more features) or your application needs a good cleanup.

Did you look at the "ps -ef" listing? Can you identify everything that is running as being valid? Can you post it here so we can look at it?
AnthonySN
Respected Contributor

Re: error in alert log.

the old system was hpux 11.11 and oracle 8i
its almost a year now since we migrated.
data has increased but the no. of oracle session was almost the same ie 1600+
We had rebooted the server one time and the memory usage was normal.
when the no of sessions in oracle touches 1600 swapinfo shows.
TTr
Honored Contributor

Re: error in alert log.

>We had rebooted the server one time and the memory usage was normal.

You need to look at the process mix and identify any runaway processes. When the users kick-off an oracle job and then disconnect, that job many times continues to run consuming resources. You need to identify and cleanup those processes.