topic Re: profile: cannot fork: too many processes in Operating System - HP-UX

profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 14:57:29 GMT

/etc/profile: cannot fork: too many processes

I am not able to login as user scott

su - scott

/etc/profile: cannot fork: too many processes
and session killed

kctune | grep maxu
maxuprc 512 512 Immed

ps -fu scott | wc -l
514

lsof -u scott | wc -l
278877
esxha309 root /home/siebel:lsof -u scott | head -10
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 4283 scott cwd DIR 128,0x2 1024 55832 /siebel/inst/siebsrvr/admin/IFB/LOG
ksh 4283 scott txt REG 64,0x8 538740 12268 /usr/bin/rksh
ksh 4283 scott mem REG 64,0x8 5020468 48 /usr/lib/hpux32/libc.so.1
ksh 4283 scott mem REG 64,0x8 293088 75 /usr/lib/hpux32/libxti.so.1
ksh 4283 scott mem REG 64,0x8 1407208 72 /usr/lib/hpux32/libnsl.so.1
ksh 4283 scott mem REG 64,0x8 76236 51 /usr/lib/hpux32/libdl.so.1
ksh 4283 scott mem REG 64,0x8 1066624 1615 /usr/lib/hpux32/dld.so
ksh 4283 scott mem REG 64,0x8 177040 70 /usr/lib/hpux32/uld.so
ksh 4283 scott 0r FIFO 0xe0000003725eb608 0t0 8772280

esxha309 root /home/siebel:ps -fu scott | wc -l
514

If I kill these scott owned processes it comes again with new pid
some script is creating them
how to find the script which creates them
most of these scott owned process are or -ksh

Please help me to kill these defunct processes . i can not increase maxuprc.

Re: profile: cannot fork: too many processes

Dennis Handly — Mon, 03 Jan 2011 15:12:40 GMT

Can you show us some example scott process trees:
UNIX95=EXTENDED_PS ps -H -fu scott

>ksh 4283 scott txt REG 64,0x8 538740 12268 /usr/bin/rksh

It seems scott may be using lots of rksh?

>If I kill these scott owned processes it comes again with new pid. some script is creating them

You are trying to logon but don't know what scott is doing?
Anyway, by looking at the hierarchy you may get an idea who is starting them.

>how to find the script which creates them
>most of these scott owned process are or -ksh
>Please help me to kill these defunct processes.

You can remove the zombies by killing the zombie master. You'll have to determine which is more important, killing the master and restarting or letting it continue.

Re: profile: cannot fork: too many processes

Mel Burslan — Mon, 03 Jan 2011 15:13:40 GMT

When you kill the process, note the PPID (parent process ID) and see if the newly spawning process has the same PPID. If so, after making sure it is not a system process and/or a critical production application, you can see what it is and kill it if it will not harm anything else.

I have seen some "clever-above-all" developers leaving a script running on their desktop terminal, doing exactly this, who don't know how to properly code a daemon. So, thread carefully but I am sure you can find the mother of all evil in this case and eliminate it.

Good luck

Re: profile: cannot fork: too many processes

Alzhy — Mon, 03 Jan 2011 15:47:43 GMT

As root:

UNIX95=1 ps -efH|tee scott.txt

And send us scott.txt

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 16:59:24 GMT

Please find output of
UNIX95=1 ps -efH|tee scott.txt

Thanks

Re: profile: cannot fork: too many processes

Alzhy — Mon, 03 Jan 2011 17:06:50 GMT

Wow... who/what is teh "scott" login? Can you kill all of its processes right now in aid of diagnosis?

for PID in `ps -ef|grep -w scott|awk '{print $2}'`;do
kill -9 $PID
done

Also, can you check if it has a cron entry? (crontab -l scott) and see if it has a script that is to spawn additional nested ksh sessions?

Re: profile: cannot fork: too many processes

Bill Hassell — Mon, 03 Jan 2011 17:10:08 GMT

Based on the most recent attachment, user scott is doing a lot of bad things (ksh starts ksh starts ksh starts ksh....) To find all the scott processes:

UNIX95=on ps -fH -u scott

Then start killing the top level processes. You don't want to increase maxuprc because user scott's activities (or scripts) are broken. This is exactly what maxuprc is supposed to do -- prevent runaway scripts or processes from using up every PID.

Re: profile: cannot fork: too many processes

Mel Burslan — Mon, 03 Jan 2011 17:12:35 GMT

It looks like, this person is running a script and spawning sub-shells to do things in parallel and these shells do not clean up properly.

Unless you are the Scott person and clean up this mess by instating proper error checking and properly killing processes when they are done doing what they are supposed to do, into the script/executable, your only option is to kill all the processes belonging to this person in one clean swoop, something like this:

ps -ef | grep scott | awk {'print $2'} | xargs kill

if plain kill do not help at the end of the pipe, you can add a "-9" to the command above.

But what I am suggesting is putting a band-aid onto a deep cut wound. The script/executable causing this behavior, needs to be fixed in the source code.

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 17:25:56 GMT

I am not able to find the script.

root /:ps -ef | grep scott | awk {'print $2'} | xargs kill -9
kill: 2426: no such process
kill: 29892: no such process
....
...so on

User has run some bad script by mistake.
I do not get machine rebooted. I need to kill the script causing the issue?

Re: profile: cannot fork: too many processes

Mel Burslan — Mon, 03 Jan 2011 17:30:58 GMT

more than likely, killing the parent process is killing all the children processes before your kill gets to them.

run

ps -fu scott

right after you run this command and see if there are any processes leftover. Please post the output here again

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 17:39:12 GMT

please find the attachment
ps -fu scott > abc.txt

Re: profile: cannot fork: too many processes

Bill Hassell — Mon, 03 Jan 2011 17:39:57 GMT

> ps -ef | grep scott | awk {'print $2'} | xargs kill -9

VERY BAD CODE.

grep and ps can cause catastrophic results, especially when combined with kill. Never use grep to look for a user. Never use kill -9 as you can damage databases and application files.

To kill all the processes owned by scott, FIRST list all of them:

ps -fu scott

Then is all of them are OK to kill, you can apply the kill (and kill -1 if necessary):

ps -fu scott | awk '{print $2}' | xargs kill
and
ps -fu scott | awk '{print $2}' | xargs kill -1

That should take care of all the processes, but just to make sure:

ps -fu scott

grep will find things (partial user names, arguments on the command line, etc) you did not intend to be found.

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 17:46:26 GMT

sure. Point is noted. Thanks for the correction.

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 17:54:19 GMT

I am not able to kill these scott owned processes
It automatically gets created even after killing all processes owned by scott.

ps -fu scott | awk '{print $2}' | xargs kill
and
ps -fu scott | awk '{print $2}' | xargs kill -1

Count always remains the same :
ps -fu scott | wc -l
514

One more question
By parent-id is 1
scott 17613 1 0 17:52:18 ? 00:00:00 -ksh
scott 10636 1 0 17:48:54 ? 00:00:00 -ksh
scott 16406 10636 0 17:51:58 ? 00:00:00 -ksh
scott 16470 16406 0 17:52:00 ? 00:00:00 -ksh
scott 16620 16470 0 17:52:02 ? 00:00:00 -ksh
-3 16619 16470 0 17:52:02 ? 00:00:00
-3 16623 16406 0 17:52:02 ? 00:00:00
-3 14613 10636 0 17:50:58 ? 00:00:00

Re: profile: cannot fork: too many processes

Mel Burslan — Mon, 03 Jan 2011 17:56:41 GMT

put this code below in a file:

vi /tmp/cleaner

{ cut below excluding this line }

#!/usr/bin/sh
ps -fu scott | while read line
do
PID=$(echo $line | awk {'print $2'}
PPID=$(echo $line | awk {'print $3'}
if [ $PPID -eq 1 ]
then
kill $PID
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]
then
kill -1 $PID
fi
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]
then
echo "Process $PID won't die"
fi
fi
done

{ cut above excluding this line }

make it executable

chmod 700 /tmp/cleaner

run it

/tmp/executable >/tmp/cleaner.out 2>&1

please post /tmp/cleaner.out here

Re: profile: cannot fork: too many processes

Dennis Handly — Mon, 03 Jan 2011 18:13:25 GMT

>ME: It seems scott may be using lots of rksh?

(lsof picked the first hardlink as the name.)

>Alzhy: Can you kill all of its processes right now in aid of diagnosis?

Much easier to do:
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)
Then change to kill -HUP then -9.

>Bill: ksh starts ksh starts ksh starts ksh

You have this with clearcase's setview.

>Then start killing the top level processes.

Right, a lot of them have init as the parent, possibly indicating they didn't log off correctly.

>I need to kill the script causing the issue?

Yes. You did that if you killed all of scott's PIDs.

>Mel: killing the parent process is killing all the children processes before your kill gets to them.

Right, that also reaps the zombies.

>please find the attachment
ps -fu scott > abc.txt

Please use this instead:
UNIX95=EXTENDED_PS ps -H -fu scott > abc.txt

It seems strange you still have lots of processes. You likely have a fork bomb with every process trying to create more, no matter which you kill.
Or it could be a problem with .profile/.kshrc since I don't see any scripts or options on any of the ksh entries.

You may want to stop them all then kill them:
kill -STOP $(UNIX95=EXTENDED_PS ps -u scott -opid=)
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)

>Bill: ps -fu scott | awk '{print $2}' | xargs kill

No need to use awk and xargs:
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)

>One more question. By parent-id is 1
>scott 17613 1 0 17:52:18 ? 00:00:00 -ksh

Right, these orphans have had their parent killed and are adopted by init.

>Mel: put this code below in a file:

My "kill -STOP" will probably be easier.

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 18:13:56 GMT

Please find cleaner.out output attached.

ps -fu scott | wc -l
514

Re: profile: cannot fork: too many processes

Dennis Handly — Mon, 03 Jan 2011 18:27:59 GMT

>Please find cleaner.out output attached.

This gives a lot of bogus output due to what Bill said about ps and grep.

Instead of doing:
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]; then
Do:
ps -p $PID > /dev/null 2>&1
if [ $? -eq 0 ]; then
Or:
kill -0 $PID 2> /dev/null
if [ $? -eq 0 ]; then

Anyway, this is moot, since you need that -STOP first.

Re: profile: cannot fork: too many processes

Alzhy — Mon, 03 Jan 2011 18:39:44 GMT

Try:

kill -9 `ps -ef|grep -w scott|awk '{print $2}'`
ps -fu scott|wc -l
ps -ef -u scott|tee scott.txt

Paste so there is no "wait" in between...

Re: profile: cannot fork: too many processes

someoneElse_1 — Mon, 03 Jan 2011 19:00:30 GMT

#!/usr/bin/sh
ps -fu scott | while read line
do
PID=$(echo $line | awk {'print $2'}
PPID=$(echo $line | awk {'print $3'}
if [ $PPID -eq 1 ]
then
kill -0 $PID 2> /dev/null
if [ $? -eq 0 ]; then
kill -1 $PID 2> /dev/null
fi
if [ $? -eq 0 ]
then
echo "Process $PID won't die"
fi
fi
done

Dennis where to put -STOP in above code
is above code ok to run