Defunct processes

[Kavita Poonia]

Hello,

There are some defunct processes running on server. I need to kill them all but when I am issuing the commands kill, kill -9, kill -15 and kill -s SIGKILL they are not getting killed. Because of these defunct processes memory utilization is very high on server, so I need to kill them all.

Please suggest how to do this ?

Thanks a lot !

Regards,
Kavita

[Michael Steele_2 · Accepted solution]

Hi Kavita:

You are very much mistaken about 'defunct' processes consumming memory, or any resources of any kind. They take up an item in the process table and the process table only. This is clearly spelled out in the 'PS' man page. Or, search the itrc with the pattern 'defunct'.

You can try killing the parent or PPID. This often works. But the source of the problem is poor coding.

[Yogeeraj_1]

hi,

Killing the defunc processes would only be soothing the problem for a short while. You need to look into the problem at the root.

are the processes associated to a specific user in all cases?

revert!
kind regards
yogeeraj

[Kavita Poonia]

Thanks a lot Michael !

Ya I did know that defunct processes do not utilize the memory but some HP guys told me that they do, so I was confused. And why generally people suggest to kill defunct processes when they do not utilize any resource.

[Kavita Poonia]

Thanks a lot Yogeeraj !

Actually I do not have any other option, instead of killing them as we have recycled all the applications which are utilizing more memory. The memory utilization reduced at that time but after sometime it has again rised.

Its Parent Process is the process with the name "wrapper". I think it is a process associated with java application and we have lots of java application running on this server.

[Bill Hassell]

You can't kill a defunct process because it is already killed. It appears in the ps listing only because it was improperly handled by it's parent. A quick Google search will confirm that thousands of defunct processes will consume no resources at all except an entry in the process table. Of course, very bad programming could cause a runaway which then fill the process table so go after the programmers and fix the broken code.

Meanwhile, memory consumption is fairly easy to see. Just ask ps to tell you:

UNIX95=1 ps -eo vsz,pid,ppid,args | sort -rn | head -20

Type this exactly as shown and the biggest (local) memory consumers will be seen. For shared memory, use this command:

ipcs -bmop | sort -rnk8

The segsz column is memory consumed by shared memory.

[Michael Steele_2]

An HP Support Guy told you That!

HP Support has been going down hill steadily and I'm very sorry to here this. They have been removing all of their valuble people and replacing them with people they found on the sidewalk and paying them 15 cents and hour.

[Kavita Poonia]

Hi Bill,

Thanks a lot for your reply. It was really helpful. The java processes are consuming lots of memory on server.

[Kavita Poonia]

Hi Michael,

Ya HP support guy told me that they consume resources and memory too, so I was confused. Anyways now its very much clear that they do not consume any memory or resource but they are not good for the system, right.

Thank you !

[Michael Steele_2]

Hi Kavita:

I'm very sorry for your experience with HP support. They're suppose to be where you go for the final answer to your problems. I have used them as references in committee meetings for years always with the comment, "...this is what the manufacturer recommends..".

Use to be you there was a philosophy to 'capture the information from the source' in IT because the source is where you find the answer. Well, hp is the source and they didn't provide you with an answer to a problem thats decades old.

As for your question, I think both Bill and I have given the correct information. 'defunct' processes only take up an entry in the process or ps table and exist due to poor programming clean up. Also, the UNIX95 command is one of my favorites. I'm glad Bill provided it. Store it away and use it daily, or when you're trouble shooting a problem box. Refer to the man page for 'ps' and note the -o arguements for a variety of reporting information. Like pcpu, vsz, rsz, time, state, etc.

[Dennis Handly]

You might want to also search the forum for zombies. In your case the zombie master is "wrapper".

>but they are not good for the system

Well, you'll need to decide whether the zombie master is worth more dead than alive, until you get it recoded.
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1241242
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1225465
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1219060
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1186347
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1145648

[Sharma Sanjeev]

Hi Kavita

If you find that process is in Defunc State and memory has not bern released you can use
IPCS to release memory.
Yes you are right if there are soo many defunc processes are there in system than they consume memory.

Regards
Sanjeev

[Dennis Handly]

>Sanjeev: memory has not ben released you can use ipcrm(1) to release memory.

I assume that zombies have already decremented the shmmem counts.

>if there are so many zombies ... they consume memory.

Only the trivial amount for the process table slots.

[Kavita Poonia]

Thanks a lot Michael !

Thanks a lot Dennis. I have gone through those forums, they were really helpful. Ya you are right wrapper is the zombie process.

Thanks a lot Sanjeev. Ya the memory is not released. I need to clear those defunct processes. But I do know the proper use of IPCS.......I mean how to use it for releasing memory which are consumed by these defunct processes.

[Bill Hassell]

Releasing shared memory segments is easy: man ipcrm

Knowing which ones are truly idle and unused is a lot harder. Shared memory segments are created by a program and then the segment information is shared with other programs. All the programs share these memory areas until they terminate. If the programs are killed improperly, none of the segments will be released.

Use the options: ipcs -bmop

Then look at NATTACH, CPID, LPID:
NATTACH = number of process attached to the segment
CPID = PID of the creator of the segment
LPID = PID of the last process to attach or detach

If NATTACH is 0 and the LPID and CPID do not point to a running process, you can probably remove the segment with ipcrm. On a running system with correctly running processes, using ipcrm can be a bit risky. If you remove the wrong segment, processes using that segment will likely crash. It would be a lot safer to reboot. Then be quite careful about using kill as root, and never use kill -9 except as a last resort. kill -9 will not allow processes to clean up properly (as in: release shared memory).

[Dennis Handly]

>Because of these defunct processes memory utilization is very high on server

Are you sure this memory use is due to the zombies? You may find there is another problem. You didn't use "kill -9" and have it work?

>Ya you are right wrapper is the zombie process.

Zombie master.

Bill: It would be a lot safer to reboot.

Then you might as well try killing the zombie master.

[Rasheed Tamton]

Hi,

I see the issue is discussed in very detail. Some times the below steps helps to find out what is going on with the zombie one

ps -el|grep defunct
truss -v all -cfp
lsof -p

Regards.

[Dennis Handly]

>Rasheed: Some times the below steps helps to find out what is going on with the zombie

tusc won't work on zombies, they are dead.
lsof -p is probably useless since the files opened by the zombie master wouldn't have much of a relation to the zombie.

[Kavita Poonia]

Hi Bill,

Thanks for you reply.

ebzwls01:/home # ipcs -bmop
IPC status from /dev/kmem as of Tue Jul 22 17:45:31 2008
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
m 0 0x411c0231 --rw-rw-rw- root root 0 348 788 13066
m 1 0x4e0c0002 --rw-rw-rw- root root 1 31040 788 13066
m 2 0x41202509 --rw-rw-rw- root root 1 8192 788 13066
m 4611 0x0c6629c9 --rw-r----- root root 2 19068672 1729 11959
m 4 0x06347849 --rw-rw-rw- root root 1 77384 1729 1829
m 1541 0x49182e06 --rw-r--r-- root root 0 22908 1723 1829
ebzwls01:/home # ps -ef|grep -i 1829
root 1829 1 0 Feb 10 ? 477:51 /opt/perf/bin/scopeux
ebzwls01:/home # ps -ef|grep -i 13066
root 13214 6494 0 17:54:20 pts/tb 0:00 grep -i 13066
ebzwls01:/home #

I have run ipcs -bmop command and the output pasted as above. There are two LPIDs for which NATTACH is 0. The LPID=1892 is running on server but LPID=13066 is not running.

Now what should I do ?

[Kavita Poonia]

Thanks Dennis for your reply.

Thanks Rasheed too........

[Dennis Handly]

>ipcs -bmop

You might want to do: ipcs -ma
Unfortunately this just gives time but not date. You can get this by compiling my attachment in this thread:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1136341

>m 4611 0x0c6629c9 --rw-r----- root root 2 19,068,672

This is the biggest but it is in use.

># ps -ef | grep -i 1829

You can simplify this by asking for PID: ps -f -p 1829 -p 13066

>There are two LPIDs for which NATTACH is 0. The LPID=1892 is running on server but LPID=13066 is not running.

These are of trivial size, you aren't going to get much memory here.

Did we ever ask for your "swapinfo -tam" output?

[Torsten.]

What process we are talking about?

Some may be related to the diagnostics and showing expected behaviour.

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1186347

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=765302

...

[Kavita Poonia]

Thanks a lot Dennis and Torsten !