Could not fork: Not enough space

[yulianto piyut]

hi all,

I have two server cluster using oracle RAC. there are errors in syslog. servers is down after these messages:

any issue in kernel parameter ?
grep from kmtune:

nproc 10240 - 10240
maxuprc 9217 Y (((NPROC*9)/10)+1)
nfile 155648 - (15*NPROC+2048)

Aug 14 23:22:17 eaipdb3 prngd[1909]: Could not fork: Not enough space
Aug 14 23:24:23 eaipdb3 syslog: Cluster Ready Services completed waiting on depe
ndencies.
Aug 14 23:24:25 eaipdb3 syslog: Duplicate spawn of Oprocd. See init.cssd for des
cription.
Aug 14 23:23:59 eaipdb3 prngd[1909]: Could not fork: Not enough space
Aug 14 23:24:25 eaipdb3 above message repeats 2 times
Aug 14 23:24:25 eaipdb3 syslog: Oprocd received graceful shutdown request. Shutt
ing down.
Aug 14 23:24:26 eaipdb3 syslog: Duplicate clsomon found. Restarting.
Aug 14 23:24:27 eaipdb3 syslog: Oracle clsomon shutdown successful.
Aug 14 23:24:27 eaipdb3 syslog: Oracle CSSD shell script failure. Duplicate CSSD

many thanks for your time to give me some suggestions and solutions!


-yut-

[whiteknight]

Hi Bantul,


You can use sar -v 1 10 to monitor the 11:46:35 text-sz ov proc-sz ov inod-sz ov file-sz ov
11:46:36 N/A N/A 133/1300 0 679/1708 0 620/2843 0
11:46:37 N/A N/A 133/1300 0 679/1708 0 620/2843 0
11:46:38 N/A N/A 133/1300 0 679/1708 0 620/2843 0
11:46:39 N/A N/A 133/1300 0 679/1708 0 620/2843 0


usually cannot fork, is your nproc is running out. Modify the value accordingly.


WK

[yulianto piyut]

hi white,

thanks for your quick response.
what value that must be configure ?

11:10:41 text-sz ov proc-sz ov inod-sz ov file-sz ov
11:10:46 N/A N/A 721/10240 0 1416/83968 0 6002/155658 0
11:10:51 N/A N/A 720/10240 0 1415/83968 0 6000/155658 0
11:10:56 N/A N/A 720/10240 0 1415/83968 0 6001/155658 0
11:11:01 N/A N/A 720/10240 0 1415/83968 0 6003/155658 0
11:11:06 N/A N/A 720/10240 0 1415/83968 0 6003/155658 0
11:11:11 N/A N/A 720/10240 0 1416/83968 0 6004/155658 0
11:11:16 N/A N/A 726/10240 0 1431/83968 0 6026/155658 0
11:11:21 N/A N/A 720/10240 0 1415/83968 0 6003/155658 0
11:11:26 N/A N/A 720/10240 0 1415/83968 0 6003/155658 0
11:11:31 N/A N/A 715/10240 0 1410/83968 0 5977/155658 0


-yut-

[A. Clay Stephenson]

You are not hitting nproc or maxupc. Fork() would set errno = EAGAIN in those cases. You are seeing ENOMEM which means you are out of virtual address space. You need to add more swap or reduce the memory footprint (possibly tune down Oracle SGA size).

[Steven E. Protter]

Shalom,

Note that if the system is stressed adding swap is merely a short term fix. You will need to add memory to feed the beast, memory hungry oracle.

Tuning the SGA is an excellent idea and if the settings are too high, trimming that can save you a great deal of money on a hardware upgrade.

You may get lucky and find a kernel parameter that needs adjustment with some overall performance measurement:

http://www.hpux.ws/system.perf.sh

Note though that a memory purchase is the most likely outcome from this problem. If the system is paging:

vmstat
swapinfo -tam

Then adding swap will merely make it run more smoothly, and probably slower than is acceptable.

I'd also make sure at some point that Oracle OS patching requirements are met and a recent bi-annual patch update from HP is installed. The oracle installer mode that oracle gives permitting bypass of system requirements should NEVER be used. I wish Oracle would pull it from their documents.

SEP

[whiteknight]

Bantul,

from your sar output, you do not have any issue on nproc, nfile.... as it is still sufficient.


From the syslog.log

Duplicate spawn of Oprocd ==> is this process behaviour abnormally ?

My suggestion you check this oracle.

WK

[yulianto piyut]

i don't know white.
concern in dba team.