Number of processes, fork problem

[Robert Milne]

Dear forum users,

When doing some load testing on our L1000 I received the following message when the 24th user logged onto our Progress database application:

[7]: The fork function failed. Too many processes already exist.

I presume this is related to a kernel parameter like nproc being too low. I had already set this up via maxusers (currently 128) to 1044. I am aiming to have no more than 55 users on our database app. I thought this was more than enough. Curiously when doing a ps -ef with 18 users logged on there was only 213 processes running. Could it be something other than nproc ?

Would anyone have any clues based on experience ? What should be a good setting (I know this can be a bit of a piece of string question...), should I just double maxusers to 256 and see what happens ? Other than Progress, LVM and the usual HP-UX 11i utilities nothing else will be ruuning on this box. It has 1.5GB of memory in it. Any advice accepted.

Thanks for your help, these forums are great !

Rob.

[Clemens van Everdingen]

Hi,

You could have a problem with swap !

Try: swapinfo -tam

If it is at or near 100% you will either need to shutdown some processes to free memory/swap, or add additional memory/swap.

C.

[Deepak Extross]

See this thread:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x35711012aa92d5118ff10090279cd0f9,00.html

[Olav Baadsvik]

Hi,

Check the kernle-parameter maxuprc
which is the max number of processes
that can be assisiated with one user

Regards
Olav

[T G Manikandan]

There are two parameters to look into.

1.maxuprc-----maximum number of processes for a user.
2.nproc--------maximum number of processes on the system

Just check whether your user is crossing the mark for the maxuprc.Also check whether some script has gone into a loop for the particular user.
If so increase the kernel parameter as reqd.

Thanks
G Manikandan

[Robert Milne]

Hello TG, Olaf and Clemens,

Current settings are:
maxuprc 75
maxusers 128
nproc 1044

I can't see any single user consuming 75 processes, but then again in my load testing I did log on many times as one user name, could this do it ? Is it processes by actual user name or does each logon, irrespective of username, constitute a separate ID ?

and swapinfo shows:
>swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 3072 0 3072 0% 0 - 1 /dev/vg00/lvol2
reserve - 594 -594
memory 1130 511 619 45%
total 4202 1105 3097 26% - 0

But this is of course un-loaded. I need to load it up again to see what happens to this.

Typically what do you guys set maxusers and nproc at ?

Thanks.

Rob.

[Tim D Fulford]

In addition to the above.

o /var/adm/syslog/syslog.log should give you some information if a kernel parameter has been breached, or even use dmesg
o glance (if you have it) gives you how full system tables are (glance -t, or just type after entering glance t)
o Some other kernel params worth checking, shmmax, maxtsiz, maxdsiz, maxssiz, npty & nstrpty

regards

Tim

[Clemens van Everdingen]

Hi,

I would try the following:

maxuprc 150
maxusers 128
nproc 1044

Test again and see what happens.

Also check syslog for reaching limits on some parameters.

C.

[John Carr_2]

Robert your error relates to the number of spawned child processes.

I had this a few years ago with an oracle application spawning lots of child processes we increased the maxuprc which simply allowed more processes to be spawned before it crashed.

We eventually resolved our problem by rewriting the application to only allow X number of child preocesses at any one time.

I think a good starting point for you is to increase the maxuprc and rebuild the kernelm, then rerun the load test.


john.

[Roger Baptiste]

hi,

You can check/monitor the usage of proc tables, either through Measureware or glance.
If realtime, run glance and option "t" . It shows the process usage. If you want to know the usage history, either run perfview and get the graphical chart

My guess is you are exceeding the maxuprc limit at a particular instant and by the time the command errors out and you verify it becomes normal.

HTH
raj

[Trond Haugen]

Does your database vendor have any rocemendations for kernel parameters?
Are you using the "standard" kernel, or one of the templates (especially the database template)?

Regards,
Trond

[Robert Milne]

Hello all,

I tried the load tests again, this time logging in as a different user each time, instead of as the same one. I think this may have been blowing the maxuprc.

Anyway I got past the 24 users this time up to 40 users, (target 55 !). Then the database started to act funny and re-started. No fork errors appeared. I had a look at this same time in the /var/adm/syslog/syslog.log as Tim suggested to see if a kernel parameter had been breached. The message:

Apr 10 14:43:28 kronehp vmunix: file: table is full
Apr 10 14:48:41 kronehp vmunix: file: table is full
Apr 10 14:55:29 kronehp above message repeats 53 times

But I am not sure which parameter this pertains to. I don't have glance or anything like that. Has anyone any clues ? I so far set the parameters according to Progress' recommendations. But have not changes anything yet since the original posting of this message.

Rob.

[Olav Baadsvik]

Hi,

Increase nfile
to get rid of the last error-message
you reported.

Regards
Olav

[Trond Haugen]

nfile for sure. If you db-vendor don't have any suggestions try doubbling it. As mentioned earlyer most db-vendors have some recomendations for kernel parameters.

regards,
Trond

[Clemens van Everdingen]

Hi,

Clear message from syslog>
Increase nfile.

C.