profile: cannot fork: too many processes

[someoneElse_1]

/etc/profile: cannot fork: too many processes




I am not able to login as user scott

su - scott

/etc/profile: cannot fork: too many processes
and session killed


kctune | grep maxu
maxuprc 512 512 Immed

ps -fu scott | wc -l
514

lsof -u scott | wc -l
278877
esxha309 root /home/siebel:lsof -u scott | head -10
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ksh 4283 scott cwd DIR 128,0x2 1024 55832 /siebel/inst/siebsrvr/admin/IFB/LOG
ksh 4283 scott txt REG 64,0x8 538740 12268 /usr/bin/rksh
ksh 4283 scott mem REG 64,0x8 5020468 48 /usr/lib/hpux32/libc.so.1
ksh 4283 scott mem REG 64,0x8 293088 75 /usr/lib/hpux32/libxti.so.1
ksh 4283 scott mem REG 64,0x8 1407208 72 /usr/lib/hpux32/libnsl.so.1
ksh 4283 scott mem REG 64,0x8 76236 51 /usr/lib/hpux32/libdl.so.1
ksh 4283 scott mem REG 64,0x8 1066624 1615 /usr/lib/hpux32/dld.so
ksh 4283 scott mem REG 64,0x8 177040 70 /usr/lib/hpux32/uld.so
ksh 4283 scott 0r FIFO 0xe0000003725eb608 0t0 8772280

esxha309 root /home/siebel:ps -fu scott | wc -l
514


If I kill these scott owned processes it comes again with new pid
some script is creating them
how to find the script which creates them
most of these scott owned process are or -ksh

Please help me to kill these defunct processes . i can not increase maxuprc.

[Dennis Handly · Accepted solution]

Can you show us some example scott process trees:
UNIX95=EXTENDED_PS ps -H -fu scott

>ksh 4283 scott txt REG 64,0x8 538740 12268 /usr/bin/rksh

It seems scott may be using lots of rksh?

>If I kill these scott owned processes it comes again with new pid. some script is creating them

You are trying to logon but don't know what scott is doing?
Anyway, by looking at the hierarchy you may get an idea who is starting them.

>how to find the script which creates them
>most of these scott owned process are or -ksh
>Please help me to kill these defunct processes.

You can remove the zombies by killing the zombie master. You'll have to determine which is more important, killing the master and restarting or letting it continue.

[Mel Burslan]

When you kill the process, note the PPID (parent process ID) and see if the newly spawning process has the same PPID. If so, after making sure it is not a system process and/or a critical production application, you can see what it is and kill it if it will not harm anything else.

I have seen some "clever-above-all" developers leaving a script running on their desktop terminal, doing exactly this, who don't know how to properly code a daemon. So, thread carefully but I am sure you can find the mother of all evil in this case and eliminate it.

Good luck

[Alzhy]

As root:

UNIX95=1 ps -efH|tee scott.txt

And send us scott.txt

[someoneElse_1]

Please find output of
UNIX95=1 ps -efH|tee scott.txt

Thanks

[Alzhy]

Wow... who/what is teh "scott" login? Can you kill all of its processes right now in aid of diagnosis?

for PID in `ps -ef|grep -w scott|awk '{print $2}'`;do
kill -9 $PID
done

Also, can you check if it has a cron entry? (crontab -l scott) and see if it has a script that is to spawn additional nested ksh sessions?

[Bill Hassell]

Based on the most recent attachment, user scott is doing a lot of bad things (ksh starts ksh starts ksh starts ksh....) To find all the scott processes:

UNIX95=on ps -fH -u scott

Then start killing the top level processes. You don't want to increase maxuprc because user scott's activities (or scripts) are broken. This is exactly what maxuprc is supposed to do -- prevent runaway scripts or processes from using up every PID.

[Mel Burslan]

It looks like, this person is running a script and spawning sub-shells to do things in parallel and these shells do not clean up properly.

Unless you are the Scott person and clean up this mess by instating proper error checking and properly killing processes when they are done doing what they are supposed to do, into the script/executable, your only option is to kill all the processes belonging to this person in one clean swoop, something like this:

ps -ef | grep scott | awk {'print $2'} | xargs kill

if plain kill do not help at the end of the pipe, you can add a "-9" to the command above.

But what I am suggesting is putting a band-aid onto a deep cut wound. The script/executable causing this behavior, needs to be fixed in the source code.

[someoneElse_1]

I am not able to find the script.

root /:ps -ef | grep scott | awk {'print $2'} | xargs kill -9
kill: 2426: no such process
kill: 29892: no such process
....
...so on

User has run some bad script by mistake.
I do not get machine rebooted. I need to kill the script causing the issue?

[Mel Burslan]

more than likely, killing the parent process is killing all the children processes before your kill gets to them.

run

ps -fu scott

right after you run this command and see if there are any processes leftover. Please post the output here again

[someoneElse_1]

please find the attachment
ps -fu scott > abc.txt

[Bill Hassell]

> ps -ef | grep scott | awk {'print $2'} | xargs kill -9

VERY BAD CODE.

grep and ps can cause catastrophic results, especially when combined with kill. Never use grep to look for a user. Never use kill -9 as you can damage databases and application files.

To kill all the processes owned by scott, FIRST list all of them:

ps -fu scott

Then is all of them are OK to kill, you can apply the kill (and kill -1 if necessary):

ps -fu scott | awk '{print $2}' | xargs kill
and
ps -fu scott | awk '{print $2}' | xargs kill -1

That should take care of all the processes, but just to make sure:

ps -fu scott

grep will find things (partial user names, arguments on the command line, etc) you did not intend to be found.

[someoneElse_1]

sure. Point is noted. Thanks for the correction.

[someoneElse_1]

I am not able to kill these scott owned processes
It automatically gets created even after killing all processes owned by scott.

ps -fu scott | awk '{print $2}' | xargs kill
and
ps -fu scott | awk '{print $2}' | xargs kill -1

Count always remains the same :
ps -fu scott | wc -l
514


One more question
By parent-id is 1
scott 17613 1 0 17:52:18 ? 00:00:00 -ksh
scott 10636 1 0 17:48:54 ? 00:00:00 -ksh
scott 16406 10636 0 17:51:58 ? 00:00:00 -ksh
scott 16470 16406 0 17:52:00 ? 00:00:00 -ksh
scott 16620 16470 0 17:52:02 ? 00:00:00 -ksh
-3 16619 16470 0 17:52:02 ? 00:00:00
-3 16623 16406 0 17:52:02 ? 00:00:00
-3 14613 10636 0 17:50:58 ? 00:00:00

[Mel Burslan]

put this code below in a file:

vi /tmp/cleaner

{ cut below excluding this line }

#!/usr/bin/sh
ps -fu scott | while read line
do
PID=$(echo $line | awk {'print $2'}
PPID=$(echo $line | awk {'print $3'}
if [ $PPID -eq 1 ]
then
kill $PID
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]
then
kill -1 $PID
fi
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]
then
echo "Process $PID won't die"
fi
fi
done

{ cut above excluding this line }

make it executable

chmod 700 /tmp/cleaner

run it

/tmp/executable >/tmp/cleaner.out 2>&1

please post /tmp/cleaner.out here

[Dennis Handly]

>ME: It seems scott may be using lots of rksh?

(lsof picked the first hardlink as the name.)

>Alzhy: Can you kill all of its processes right now in aid of diagnosis?

Much easier to do:
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)
Then change to kill -HUP then -9.

>Bill: ksh starts ksh starts ksh starts ksh

You have this with clearcase's setview.

>Then start killing the top level processes.

Right, a lot of them have init as the parent, possibly indicating they didn't log off correctly.

>I need to kill the script causing the issue?

Yes. You did that if you killed all of scott's PIDs.

>Mel: killing the parent process is killing all the children processes before your kill gets to them.

Right, that also reaps the zombies.

>please find the attachment
ps -fu scott > abc.txt

Please use this instead:
UNIX95=EXTENDED_PS ps -H -fu scott > abc.txt

It seems strange you still have lots of processes. You likely have a fork bomb with every process trying to create more, no matter which you kill.
Or it could be a problem with .profile/.kshrc since I don't see any scripts or options on any of the ksh entries.

You may want to stop them all then kill them:
kill -STOP $(UNIX95=EXTENDED_PS ps -u scott -opid=)
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)

>Bill: ps -fu scott | awk '{print $2}' | xargs kill

No need to use awk and xargs:
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)

>One more question. By parent-id is 1
>scott 17613 1 0 17:52:18 ? 00:00:00 -ksh

Right, these orphans have had their parent killed and are adopted by init.

>Mel: put this code below in a file:

My "kill -STOP" will probably be easier.

[someoneElse_1]

Please find cleaner.out output attached.

ps -fu scott | wc -l
514

[Dennis Handly]

>Please find cleaner.out output attached.

This gives a lot of bogus output due to what Bill said about ps and grep.

Instead of doing:
ps -ef | grep $PID; r=$?
if [ $r -eq 0 ]; then
Do:
ps -p $PID > /dev/null 2>&1
if [ $? -eq 0 ]; then
Or:
kill -0 $PID 2> /dev/null
if [ $? -eq 0 ]; then

Anyway, this is moot, since you need that -STOP first.

[Alzhy]

Try:

kill -9 `ps -ef|grep -w scott|awk '{print $2}'`
ps -fu scott|wc -l
ps -ef -u scott|tee scott.txt

Paste so there is no "wait" in between...

[someoneElse_1]

#!/usr/bin/sh
ps -fu scott | while read line
do
PID=$(echo $line | awk {'print $2'}
PPID=$(echo $line | awk {'print $3'}
if [ $PPID -eq 1 ]
then
kill -0 $PID 2> /dev/null
if [ $? -eq 0 ]; then
kill -1 $PID 2> /dev/null
fi
if [ $? -eq 0 ]
then
echo "Process $PID won't die"
fi
fi
done


Dennis where to put -STOP in above code
is above code ok to run

[Bill Hassell]

Since the problem seems to be recurring, look at /etc/inittab to see if someone put a bad entry there. A respawn for this problem script may be causing all this mess. Change respawn to off, comment the entry or simply remove the entry. Or the problem may be in cron -- look at both root and scott cron entries. And finally, remove any bad profile code by renaming ~scott/.profile and ~scott/.kshrc to something else like .profile.old and .kshrc.old.

[someoneElse_1]

.profile and .kshrc is being shared with other user and that user is not having any issue.
cat /etc/passwd | grep scott
scott:*:20445:20244:Application Developers:/home/scott:/usr/bin/ksh

crontab has no entry for scott user

inittab content attached

[Alzhy]

Look at ~/scott/.ssh
If it exists, rename authorized_keys or authorized_keys2 file to something like *.org

Long shot..but scott's processes could be spawned via ssh?

[Dennis Handly]

>where to put -STOP in above code?
>is above code ok to run

You don't. You can run the above but you won't get anywhere with the fork bomb. My suggestions on Mel's scripts are to prevent useless output. But they don't solve anything.

You need to do these two steps. The first to stop the slippery PIDs and the second to kill them all. You could repeat the first several times.
kill -STOP $(UNIX95=EXTENDED_PS ps -u scott -opid=)
kill $(UNIX95=EXTENDED_PS ps -u scott -opid=)

>Bill: Since the problem seems to be recurring, look at /etc/inittab

A fork bomb recurs too.

>And finally, remove any bad profile code by renaming ~scott/.profile and ~scott/.kshrc to something else like .profile.old and .kshrc.old.

Yes, I was also thinking that but just changing the permissions to: ugo=
I.e. after we kill all of scott, the next login may start it again.

>.profile and .kshrc is being shared with other user and that user is not having any issue.

How is it shared, a copy?
Why not change the permissions and see if things stop?

>inittab content attached

I don't see it.

>Alzhy: scott's processes could be spawned via ssh?

>rename authorized_keys or authorized_keys2 file

Instead of rename, change the permissions: chmod u-rwx ~scott/.ssh

The following may tell you the login source:
who -a | grep scott

[someoneElse_1]

thanks I am able to login now

kill -STOP $(UNIX95=EXTENDED_PS ps -u scott -opid=)

I ran this 3 times
and all processes killed

now i can login with user scott


Dennis ThankYou very much for help.

This is very helpful