error in alert log.

[AnthonySN]

we have a 2 node cluster with OS hpux 11.31,mc/sg and oracle 10g
we are getting the below errors in the alert log and the application needs to be restarted because it hangs.

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed


tail -20 alert_2K.log
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:47 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:47 2010
ksvcreate: Process(q001) creation failed
Tue Feb 16 11:07:58 2010
Process startup failed, error stack:
Tue Feb 16 11:07:58 2010
Errors in file ...._psp0_22791.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed

[TTr]

It looks like you have hit a system limit and can your oracle account can not start up anymore processes.

Check you /var/adm/syslog/syslog.com for more information. You may have to increase some kernel parameters such as nproc, maxuprcs, nfile, nflocks etc. Also check you memory and swap space utilization.

[Dennis Handly]

>TTr: Also check your memory and swap space utilization.

Right, you want to do a "swapinfo -tam" at the same time as your heavy load, to be as accurate as possible.

[Kapil Jha]

Is it a new setup and if yes have u tunned various kernel parameters in the server.

Plus check your memory usage when you trying to start the oracle.

BR,
Kapil+

[Horia Chirculescu]

Hello, Sasj,

Please read the following document related to recommended kernel parameters:

http://h21007.www2.hp.com/portal/download/files/unprot/database/HP3KOracle.ppt

Make the settings and then try to start again your instance. If still have problems, should check also oracle initialization parameters

Best regards,
Horia.

[AnthonySN]

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 380 16004 2% 0 - 1 /dev/vg00/lvol2
reserve - 16002 -16002
memory 15555 15247 308 98%
total 31939 31629 310 99% - 0 -

We have noticed that there is no swapping even when the memory reaches 100% utilization.
also changed nproc value. and have reduced the filecache_max/min to 5 & 3 % respectively.

[Steven Schweda]

> [...] fork failed with status: 11

Around here, error 11 is EAGAIN, "No more
processes". I'd worry less about memory,
and more about a limit on process count. Of
course, the error code could be misleading.

> [...] also changed nproc value.

From what to what?

man ulimit

[AnthonySN]

from 4200 to 5000

[Horia Chirculescu]

Can you check for Bug 3411021 Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 ?

It seems that this bug is fixed on 10.2.0.3+.

What is your version of oracle?

Horia.

[AnthonySN]

oracle 10gR2.

the problem occurs only when the swapinfo shows 100% utilization of memory.
ram and swap is 16GB each.
why is there no swapping.

[TTr]

> reserve - 16002 -16002

It looks like your address space may have been exhausted.
There may not be active disk swapping which means that you will not have i/o issues but almost all your swap space is reserved by the running processes.

You need to add more swap space.

After you add more swap space, more processes will run which means that the server most likely will swap to disk actively and the performance will decrease tremendously. So adding more memory would be a good idea.

[AnthonySN]

TTr> but almost all your swap space is reserved by the running processes.

which processes you mean, which is that process which requires 16GB swap reservation.

[TTr]

> which processes you mean, which is that process which requires 16GB swap reservation

All of the running processes combined.

In the meantime you can look in the running processes and see if you can clean up any of them or turn off any unnecessary services.

[AnthonySN]

the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram it would be not justifiable to ask for more physical memory.

[AnthonySN]

TTr>>
> reserve - 16002 -16002
It looks like your address space may have been exhausted.

may you pls shed some light on the above.
is there any way to reduce the reserve.
how can the d/b start using swap space if short of memory. and what is address space ?

[Dennis Handly · Accepted solution]

>is there any way to reduce the reserve?

Only by running fewer processes, or smaller/fewer shared memory segments.

>how can the db start using swap space if short of memory.

You have to have more unreserved device swap.

>and what is address space?

In this case, I assume TTr really meant reserved device swap used up all of the device swap.

[AnthonySN]

> Only by running fewer processes, or smaller/fewer shared memory segments.

how can we reduce those processes?
we have checked the services running and they are bare minimum, its a cluster and ora10g running on it, whenever the mumber of sessions reaches 1600+ the memory utilization touches 100%.
and why is that swap utilization shows only 3% used.
we need strong reasons to justify the additional memory needed.

[Dennis Handly]

>the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram

What does "swapinfo -tam" show for both machines?
Did you configure your Oracle SGA too large on your new machine?

[Dennis Handly]

>how can we reduce those processes?

By killing them or not starting them. Or setting some configuration value to not do so many.

>number of sessions reaches 1600+ the memory utilization touches 100%.

Less sessions?

>why is that swap utilization shows only 3% used.

You have reserved that space for growth. If you don't want to grow, you shouldn't reserve so much.

>we need strong reasons to justify the additional memory needed.

You can just add more device swap first and check your performance.

[AnthonySN]

the application is now migrated to this new system so i cant get the swapinfo -tam output now.

sga is 2.5 gb

[AnthonySN]

>Or setting some configuration value to not do so many.
have changed nproc, what else do you suggest.

>Less sessions?
cant be done since this application is used by all over the branches.

>You have reserved that space for growth. If you don't want to grow, you shouldn't reserve so much.
How do i unreserve.

[Dennis Handly]

>so I cant get the swapinfo -tam output now.

Is your SGA value larger now?

You may want to look at these recent threads and see how to use ps(1) to see how much VM is being used by each process:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1411399

[TTr]

> the application is now migrated to this new system

Did you also upgrade the oracle version? And/or the application version?

With the information given so far, there is nothing wrong with your setup. Either your application environment has grown (much more data, larger database, more users, more features) or your application needs a good cleanup.

Did you look at the "ps -ef" listing? Can you identify everything that is running as being valid? Can you post it here so we can look at it?

[AnthonySN]

the old system was hpux 11.11 and oracle 8i
its almost a year now since we migrated.
data has increased but the no. of oracle session was almost the same ie 1600+
We had rebooted the server one time and the memory usage was normal.
when the no of sessions in oracle touches 1600 swapinfo shows.

[TTr]

>We had rebooted the server one time and the memory usage was normal.

You need to look at the process mix and identify any runaway processes. When the users kick-off an oracle job and then disconnect, that job many times continues to run consuming resources. You need to identify and cleanup those processes.