topic Re: Fork function failed. in Operating System - HP-UX

Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 16:42:47 GMT

My hp ux machine 11 os gives me this message every now and then - often enough that it needs to be dealt with. Its occurrence is during a Monday (after the weekend or after a long vacation).
"The fork function failed. Too many processes already exist."

I've called HP support and they've asked me to change a few configurable kernel parameters. As of now, nproc is 1024; maxuser = 124; and maxuprc = 200. This was changed a week ago. But today, I had the same problem again. When this happens, my only choice is to do a transfer of control to reboot everything, once that's done everything seems to be fine. Any input will be appreciated. Thanks.
Kirk

Re: Fork function failed.

Ken Hubnik_2 — Mon, 06 Jan 2003 16:51:16 GMT

Have you tried increasing nprocs in the kernel??

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 16:52:47 GMT

Yes, the value was doubled from last time.

Re: Fork function failed.

Ken Hubnik_2 — Mon, 06 Jan 2003 16:57:51 GMT

May need to also bump up your nfile parameter. If you have glance you should be able to look at these usages.

Re: Fork function failed.

Chris Wilshaw — Mon, 06 Jan 2003 17:03:23 GMT

If you look at the values in sar -v 1 10, you should see something similar to

16:57:54 text-sz ov proc-sz ov inod-sz ov file-sz ov
16:57:55 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:57:56 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:57:57 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:57:58 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:57:59 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:58:00 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:58:01 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:58:02 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:58:03 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0
16:58:04 N/A N/A 556/1024 0 2800/2800 0 1532/5110 0

The proc-sz column shows the system-wide number of processes that are running. Is this close to your limit of 1024?

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 17:21:24 GMT

This is what mine looks like.

12:08:50 text-sz ov proc-sz ov inod-sz ov file-sz ov
12:08:51 N/A N/A 143/1064 0 1356/1356 0 424/2328 0
12:08:52 N/A N/A 143/1064 0 1356/1356 0 424/2328 0
12:08:53 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:54 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:55 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:56 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:57 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:58 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:08:59 N/A N/A 143/1064 0 1354/1356 0 424/2328 0
12:09:00 N/A N/A 143/1064 0 1354/1356 0 424/2328 0

Re: Fork function failed.

erics_1 — Mon, 06 Jan 2003 17:32:16 GMT

Kirk,

The important part to remember from your output is whether it is from a time when the system was producing the errors or at a time when the system was less busy. Based on your error, I'd say that nproc needs to be increased yet.

Hope this helps!
Eric

Re: Fork function failed.

Byron Myers — Mon, 06 Jan 2003 17:35:03 GMT

Kirk, also take a peek at the maxuprc kernel parm - max processes per user. I believe the default is 50. Also, you will probably get many more responses if you assign points more often to those that respond to your questions.

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 17:39:09 GMT

Are you saying to decrease my maxuprc from 200 to 50? Also, just want to let you know that there are no processes that run during the weekend, and nobody usually works during those days.

Re: Fork function failed.

Byron Myers — Mon, 06 Jan 2003 17:59:42 GMT

Kirk, NO - sorry, I did not notice that you already stated that maxuprc=200.

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 18:05:43 GMT

Also, just fyi...nfile = 2232

Re: Fork function failed.

Brian Watkins — Mon, 06 Jan 2003 19:06:40 GMT

Give this a try when the error message is occurring on the problem system:

1. Start glance
2. Look at the System Tables Report (shortcut to this is the "t" key)

Are any of these paramters at 80% utilization or higher? If so, you should consider making a backup of your current kernel and increasing the affected values by at least 10 or 20% and rebooting.

After the reboot, repeat the steps above to see what the utilization levels are. Continue to monitor them, especially during peak processing hours when the errors have been happening.

Hope this helps!

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 20:51:08 GMT

Mmm...I'm not familiar with glance, but do I start it by typing "glance" at the command line. If that's the way, then I don't have it. Anyway, I would like to increase my nproc. What is the maximum nproc that I can use? I don't want to go over it and cause my system to crash. Thanks again.

Re: Fork function failed.

Byron Myers — Mon, 06 Jan 2003 21:06:57 GMT

Kirk, Also watch for zombie processes. run "ps -ef" and look for any processes that have "" under the "COMMAND" column. Sometimes a program starts cranking out these defunct processes until the UNIX process table fills up. If you see a lot of these defunct processes, say in the hundreds, then look for their parent process in the "ps -ef" output. Kill the parent and the defunc processes go away, free-ing up the slots in the process table. The process table size is defined by nproc. If you see this behavior, that parent program has a bug in it that needs to be addressed.

Re: Fork function failed.

Kirk Solano — Mon, 06 Jan 2003 21:15:05 GMT

I do seem to have alot of those defunc proces...more than 100. What command do I use to kill them? Also, does rebooting the system clear up the process table? Because I rebooted it this morning and the output I'm looking has this defunc processes. Does this mean I have a bug somewhere?

Re: Fork function failed.

Frank Slootweg — Tue, 07 Jan 2003 12:48:21 GMT

*Where* (i.e. on your terminal? in a file?) do you get that message ("The fork function failed. Too many processes already exist." and from which command(s)?

If you get it on your terminal, then also look in your /var/adm/syslog/syslog.log file for a similar/related message of the *same time*.

If syslog.log says "... proc: table is full", then the cause is a too low nproc. It is says something else (please post message) then the cause is probably a too low maxuprc.

Re: Fork function failed.

Trond Haugen — Tue, 07 Jan 2003 13:19:40 GMT

Seems your problem is the defunct proceses. You get a "zombi" when a parent process don't "wait" for it's child. Usually a resoule of bad programming. The defunct process is nothing more than a entry in the process list. But it will hold that entry and as you have experienced it will run full. The only way to clear it is to reboot.
Finding the program that leavs the zombies will be hard "detective" work. Maybe doing some 'ps -ef' and sending to a file will help you backtrack the PID and PPID when you find the zombies.
Good luck and do assign points.

Regards,
Tron

Re: Fork function failed.

Raynald Boucher — Tue, 07 Jan 2003 14:27:43 GMT

First Review your kernel parameters as suggested above.

Then try to identify what causes the "fork bomb". It could be a script that restarts itself, a bad loop or even a typo.

We experienced this a little while ago: one of our developpers cd'd to a directory containing copybooks and typed "" instead of "cat ".
Unfortunately, all the files in the directory contained many asterisks (comments) and had execute permissions. So the shell expanded the first "*" into execute "all files in this directory" and so on.

We removed execute permissions on the files in text directories and haven't had the problem since.

Hope this helped in your diagnostic efforts.