The fork function failed. Too many processes already exist

[Stefan Schulz]

Hi,

i have a server which is displaying this message whenever i try to issue a command. This server is a D-class running HP-UX 10.20.

I know about the kernel parameters to tune. Also know that i have to look for defunct processes. The problem is i can't look for any processes because the only commant i could issue successfully was "pwd".

I need to kill some processes so that i'm able to examine the situation. What can i do?

i tried to free the process tabel by using "lpshut" and "/sbin/init.d/samba stop". But both commands died with the message "The fork function failed. Too many processes already exist".

I know that i could try to use exec command. But this will kill my last existing shell on this server. I don't want to risk to be shut out of this server still no space in the process table.

Any hints? Ist there anything i can stop without getting this error message?

Any help is really appreciated.

Regards Stefan

[Kent Ostby]

Stefan --

Do you have a shell logged in as root ?

If so, try to run a ps command and get some process IDs.

Then obviously trying to kill some of those would be good.

Do you know of any PCs where people may have defunct sessions logged in ? Ask those people to kill the connections on the PC side (kill the Ap or the telent window or whatever).

The other thing you may have to consider is using your last good shell to do a shutdown or reboot.

Best regards,

Kent M. Ostby

[Massimo Bianchi]

Hi,
go to the console, as a first step.


THen try using the TSM, so that, in case, you have always a terminal.

Thing you can try:

shutting down non-vital services, like mwa, opc, cron , sendmail, and so on.


YOu can issue the pwd because it's a built-in command, in many sheels.

CAn you ask some user to log-out, to free resources ?

If there are applications connected, are you able to disconnect some of them ?

A demy-disruptive test could be to unplug ethernet cables, to let some process die.

tcp connections will continue, maybe, after that.


HTH,
Massimo

[Stefan Schulz]

Hi Kent,

unfortunately i am not able to start something like ps -ef on my last available shell on the server.

As mentioned i know about this exec command, but if it fails i will lose my last shell without any effect.

It seems that all i can do is to try to do a clean reboot with my last shell. What would you suggest:

exec shutdown -r -y 0 OR
exec reboot

Regards Stefan

[John Palmer]

Hi Stefan,

You wont be able to run anything that requires a new process to be forked. Unfortunately, this includes the ps command.

The first thing I'd try is to get users to log out.

If that doesn't work you'll have to resort to killing arbitrary process numbers, 'kill' will work because it's built-in to the shell.

Hopefully, once you've killed one process you should be able to run 'ps' to find out what's going on.

It is possible though that you have a 'rogue' process that is continuing to attempt to fork more and if you can't identify it you'll have no option but to reboot.

Good luck,
John

[John Palmer · Accepted solution]

There's no point in running 'shutdown' because it will attempt to run many processes.

'exec reboot' should work.

Regards,
John

[Stefan Schulz]

Hi Massimo,

the last shell i have is on the console, allthough im actually have CDE running. Im afraid if i kill CDE i will not be able to use the normal terminal login.

I don't know what is eating up all my resources on this server, so this could still be an ongouing process. If i free some resources by killing CDE this resources might be eaten up immediately.

BTW: i know that there is a key combination that will kill CDE but i can't remember at the moment.

This server is a file- (nfs and cifs) and printserver. There are no other applications running. Only 20 of normally 70 are active at the moment.

So you say i could shutdown the remaining 20 clients to free up some resources? I will try this befor trying to shutdown.

Tahnks

[Muthukumar_5]

I hope your kenrel parameters process id allocations and others are very less. You have to check and recompile your kernel to support more number process ids.

If you want to check the performace try to boot on single user more and check there.

If you reboot your systems all ZOMBIE process which occupied process table will be also killed.

Same discussion but for 11.00 avaiable over,
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=107824

[Massimo Bianchi]

Hi,
in your situation i will advise users in any case, just because they may end up working , and not being capable of saving the files changed.

Every smb user will fork a connection, and a process, if I remember well.

Dangerous situation...

If you are going to shutdown, consider a TOC, to dump the memory content and have it analyzed by HP, opening a call.



REgards,
Massimo

[Dietmar Konermann]

Stefan,

since you can't fork() I believe using exec is the only way to go. So you have only one shot. If that fails you could still TOC the system to get the dump analysed later.

You already tried to halt Samba... You could ry this instead:

# exec fuser -k /opt/samba/bin/smbd /opt/samba/bin/nmb

This should free up enough slots in the proc table. Another candidate could be:

# exec fuser -k /usr/sbin/nfsd

Best regards...
Dietmar.

[Stefan Schulz]

Thanks to all your help. Because of a limited timewindow and the need get the server back in a stable status i had to use my last shell to issue an "exec reboot"

This worked and the server is up and running again.

I also tried to disconnect the clients to free up some resources, but this didn't work.

Hopefully there are some traces left so that i can figure out what happend.

Thanks again for your help. I will assign points right after this post.

Regards Stefan