topic Re: error in alert log. in Operating System - HP-UX

error in alert log.

AnthonySN — Sun, 28 Feb 2010 08:18:28 GMT

we have a 2 node cluster with OS hpux 11.31,mc/sg and oracle 10g
we are getting the below errors in the alert log and the application needs to be restarted because it hangs.

ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed

tail -20 alert_2K.log
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:47 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:47 2010
ksvcreate: Process(q001) creation failed
Tue Feb 16 11:07:58 2010
Process startup failed, error stack:
Tue Feb 16 11:07:58 2010
Errors in file ...._psp0_22791.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Tue Feb 16 11:07:59 2010
Process q001 died, see its trace file
Tue Feb 16 11:07:59 2010
ksvcreate: Process(q001) creation failed

Re: error in alert log.

TTr — Sun, 28 Feb 2010 14:09:10 GMT

It looks like you have hit a system limit and can your oracle account can not start up anymore processes.

Check you /var/adm/syslog/syslog.com for more information. You may have to increase some kernel parameters such as nproc, maxuprcs, nfile, nflocks etc. Also check you memory and swap space utilization.

Re: error in alert log.

Dennis Handly — Sun, 28 Feb 2010 22:51:36 GMT

>TTr: Also check your memory and swap space utilization.

Right, you want to do a "swapinfo -tam" at the same time as your heavy load, to be as accurate as possible.

Re: error in alert log.

Kapil Jha — Mon, 01 Mar 2010 02:42:40 GMT

Is it a new setup and if yes have u tunned various kernel parameters in the server.

Plus check your memory usage when you trying to start the oracle.

BR,
Kapil+

Re: error in alert log.

Horia Chirculescu — Mon, 01 Mar 2010 06:50:42 GMT

Hello, Sasj,

Please read the following document related to recommended kernel parameters:

http://h21007.www2.hp.com/portal/download/files/unprot/database/HP3KOracle.ppt

Make the settings and then try to start again your instance. If still have problems, should check also oracle initialization parameters

Best regards,
Horia.

Re: error in alert log.

AnthonySN — Mon, 01 Mar 2010 13:28:41 GMT

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 380 16004 2% 0 - 1 /dev/vg00/lvol2
reserve - 16002 -16002
memory 15555 15247 308 98%
total 31939 31629 310 99% - 0 -

We have noticed that there is no swapping even when the memory reaches 100% utilization.
also changed nproc value. and have reduced the filecache_max/min to 5 & 3 % respectively.

Re: error in alert log.

Steven Schweda — Mon, 01 Mar 2010 13:38:22 GMT

> [...] fork failed with status: 11

Around here, error 11 is EAGAIN, "No more
processes". I'd worry less about memory,
and more about a limit on process count. Of
course, the error code could be misleading.

> [...] also changed nproc value.

From what to what?

man ulimit

Re: error in alert log.

AnthonySN — Mon, 01 Mar 2010 13:45:00 GMT

from 4200 to 5000

Re: error in alert log.

Horia Chirculescu — Mon, 01 Mar 2010 13:53:05 GMT

Can you check for Bug 3411021 Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 ?

It seems that this bug is fixed on 10.2.0.3+.

What is your version of oracle?

Horia.

Re: error in alert log.

AnthonySN — Mon, 01 Mar 2010 14:03:31 GMT

oracle 10gR2.

the problem occurs only when the swapinfo shows 100% utilization of memory.
ram and swap is 16GB each.
why is there no swapping.

Re: error in alert log.

TTr — Mon, 01 Mar 2010 14:03:58 GMT

> reserve - 16002 -16002

It looks like your address space may have been exhausted.
There may not be active disk swapping which means that you will not have i/o issues but almost all your swap space is reserved by the running processes.

You need to add more swap space.

After you add more swap space, more processes will run which means that the server most likely will swap to disk actively and the performance will decrease tremendously. So adding more memory would be a good idea.

Re: error in alert log.

AnthonySN — Mon, 01 Mar 2010 14:07:19 GMT

TTr> but almost all your swap space is reserved by the running processes.

which processes you mean, which is that process which requires 16GB swap reservation.

Re: error in alert log.

TTr — Mon, 01 Mar 2010 14:35:01 GMT

> which processes you mean, which is that process which requires 16GB swap reservation

All of the running processes combined.

In the meantime you can look in the running processes and see if you can clean up any of them or turn off any unnecessary services.

Re: error in alert log.

AnthonySN — Tue, 02 Mar 2010 08:00:45 GMT

the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram it would be not justifiable to ask for more physical memory.

Re: error in alert log.

AnthonySN — Tue, 02 Mar 2010 08:26:57 GMT

TTr>>
> reserve - 16002 -16002
It looks like your address space may have been exhausted.

may you pls shed some light on the above.
is there any way to reduce the reserve.
how can the d/b start using swap space if short of memory. and what is address space ?

Re: error in alert log.

Dennis Handly — Tue, 02 Mar 2010 09:04:32 GMT

>is there any way to reduce the reserve?

Only by running fewer processes, or smaller/fewer shared memory segments.

>how can the db start using swap space if short of memory.

You have to have more unreserved device swap.

>and what is address space?

In this case, I assume TTr really meant reserved device swap used up all of the device swap.

Re: error in alert log.

AnthonySN — Tue, 02 Mar 2010 09:14:28 GMT

> Only by running fewer processes, or smaller/fewer shared memory segments.

how can we reduce those processes?
we have checked the services running and they are bare minimum, its a cluster and ora10g running on it, whenever the mumber of sessions reaches 1600+ the memory utilization touches 100%.
and why is that swap utilization shows only 3% used.
we need strong reasons to justify the additional memory needed.

Re: error in alert log.

Dennis Handly — Tue, 02 Mar 2010 09:20:16 GMT

>the same application was running on an old server with just 5 GB ram and hardly there were any such memory problem, with 16 GB ram

What does "swapinfo -tam" show for both machines?
Did you configure your Oracle SGA too large on your new machine?

Re: error in alert log.

Dennis Handly — Tue, 02 Mar 2010 09:23:20 GMT

>how can we reduce those processes?

By killing them or not starting them. Or setting some configuration value to not do so many.

>number of sessions reaches 1600+ the memory utilization touches 100%.

Less sessions?

>why is that swap utilization shows only 3% used.

You have reserved that space for growth. If you don't want to grow, you shouldn't reserve so much.

>we need strong reasons to justify the additional memory needed.

You can just add more device swap first and check your performance.

Re: error in alert log.

AnthonySN — Tue, 02 Mar 2010 09:23:40 GMT

the application is now migrated to this new system so i cant get the swapinfo -tam output now.

sga is 2.5 gb