topic Re: Defunct processes in Operating System - HP-UX

Defunct processes

Kavita Poonia — Sun, 20 Jul 2008 17:08:16 GMT

Hello,

There are some defunct processes running on server. I need to kill them all but when I am issuing the commands kill, kill -9, kill -15 and kill -s SIGKILL they are not getting killed. Because of these defunct processes memory utilization is very high on server, so I need to kill them all.

Please suggest how to do this ?

Thanks a lot !

Regards,
Kavita

Re: Defunct processes

Michael Steele_2 — Sun, 20 Jul 2008 17:18:09 GMT

Hi Kavita:

You are very much mistaken about 'defunct' processes consumming memory, or any resources of any kind. They take up an item in the process table and the process table only. This is clearly spelled out in the 'PS' man page. Or, search the itrc with the pattern 'defunct'.

You can try killing the parent or PPID. This often works. But the source of the problem is poor coding.

Re: Defunct processes

Yogeeraj_1 — Sun, 20 Jul 2008 17:24:29 GMT

hi,

Killing the defunc processes would only be soothing the problem for a short while. You need to look into the problem at the root.

are the processes associated to a specific user in all cases?

revert!
kind regards
yogeeraj

Re: Defunct processes

Kavita Poonia — Sun, 20 Jul 2008 17:50:34 GMT

Thanks a lot Michael !

Ya I did know that defunct processes do not utilize the memory but some HP guys told me that they do, so I was confused. And why generally people suggest to kill defunct processes when they do not utilize any resource.

Re: Defunct processes

Kavita Poonia — Sun, 20 Jul 2008 17:59:03 GMT

Thanks a lot Yogeeraj !

Actually I do not have any other option, instead of killing them as we have recycled all the applications which are utilizing more memory. The memory utilization reduced at that time but after sometime it has again rised.

Its Parent Process is the process with the name "wrapper". I think it is a process associated with java application and we have lots of java application running on this server.

Re: Defunct processes

Bill Hassell — Sun, 20 Jul 2008 19:30:58 GMT

You can't kill a defunct process because it is already killed. It appears in the ps listing only because it was improperly handled by it's parent. A quick Google search will confirm that thousands of defunct processes will consume no resources at all except an entry in the process table. Of course, very bad programming could cause a runaway which then fill the process table so go after the programmers and fix the broken code.

Meanwhile, memory consumption is fairly easy to see. Just ask ps to tell you:

UNIX95=1 ps -eo vsz,pid,ppid,args | sort -rn | head -20

Type this exactly as shown and the biggest (local) memory consumers will be seen. For shared memory, use this command:

ipcs -bmop | sort -rnk8

The segsz column is memory consumed by shared memory.

Re: Defunct processes

Michael Steele_2 — Sun, 20 Jul 2008 19:33:10 GMT

An HP Support Guy told you That!

HP Support has been going down hill steadily and I'm very sorry to here this. They have been removing all of their valuble people and replacing them with people they found on the sidewalk and paying them 15 cents and hour.

Re: Defunct processes

Kavita Poonia — Sun, 20 Jul 2008 20:01:55 GMT

Hi Bill,

Thanks a lot for your reply. It was really helpful. The java processes are consuming lots of memory on server.

Re: Defunct processes

Kavita Poonia — Sun, 20 Jul 2008 20:08:57 GMT

Hi Michael,

Ya HP support guy told me that they consume resources and memory too, so I was confused. Anyways now its very much clear that they do not consume any memory or resource but they are not good for the system, right.

Thank you !

Re: Defunct processes

Michael Steele_2 — Sun, 20 Jul 2008 21:53:55 GMT

Hi Kavita:

I'm very sorry for your experience with HP support. They're suppose to be where you go for the final answer to your problems. I have used them as references in committee meetings for years always with the comment, "...this is what the manufacturer recommends..".

Use to be you there was a philosophy to 'capture the information from the source' in IT because the source is where you find the answer. Well, hp is the source and they didn't provide you with an answer to a problem thats decades old.

As for your question, I think both Bill and I have given the correct information. 'defunct' processes only take up an entry in the process or ps table and exist due to poor programming clean up. Also, the UNIX95 command is one of my favorites. I'm glad Bill provided it. Store it away and use it daily, or when you're trouble shooting a problem box. Refer to the man page for 'ps' and note the -o arguements for a variety of reporting information. Like pcpu, vsz, rsz, time, state, etc.

Re: Defunct processes

Dennis Handly — Mon, 21 Jul 2008 00:51:27 GMT

You might want to also search the forum for zombies. In your case the zombie master is "wrapper".

>but they are not good for the system

Well, you'll need to decide whether the zombie master is worth more dead than alive, until you get it recoded.
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1241242
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1225465
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1219060
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1186347
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1145648

Re: Defunct processes

Sharma Sanjeev — Mon, 21 Jul 2008 05:18:22 GMT

Hi Kavita

If you find that process is in Defunc State and memory has not bern released you can use
IPCS to release memory.
Yes you are right if there are soo many defunc processes are there in system than they consume memory.

Regards
Sanjeev

Re: Defunct processes

Dennis Handly — Mon, 21 Jul 2008 05:57:05 GMT

>Sanjeev: memory has not ben released you can use ipcrm(1) to release memory.

I assume that zombies have already decremented the shmmem counts.

>if there are so many zombies ... they consume memory.

Only the trivial amount for the process table slots.

Re: Defunct processes

Kavita Poonia — Mon, 21 Jul 2008 19:11:40 GMT

Thanks a lot Michael !

Thanks a lot Dennis. I have gone through those forums, they were really helpful. Ya you are right wrapper is the zombie process.

Thanks a lot Sanjeev. Ya the memory is not released. I need to clear those defunct processes. But I do know the proper use of IPCS.......I mean how to use it for releasing memory which are consumed by these defunct processes.

Re: Defunct processes

Bill Hassell — Tue, 22 Jul 2008 01:31:41 GMT

Releasing shared memory segments is easy: man ipcrm

Knowing which ones are truly idle and unused is a lot harder. Shared memory segments are created by a program and then the segment information is shared with other programs. All the programs share these memory areas until they terminate. If the programs are killed improperly, none of the segments will be released.

Use the options: ipcs -bmop

Then look at NATTACH, CPID, LPID:
NATTACH = number of process attached to the segment
CPID = PID of the creator of the segment
LPID = PID of the last process to attach or detach

If NATTACH is 0 and the LPID and CPID do not point to a running process, you can probably remove the segment with ipcrm. On a running system with correctly running processes, using ipcrm can be a bit risky. If you remove the wrong segment, processes using that segment will likely crash. It would be a lot safer to reboot. Then be quite careful about using kill as root, and never use kill -9 except as a last resort. kill -9 will not allow processes to clean up properly (as in: release shared memory).

Re: Defunct processes

Dennis Handly — Tue, 22 Jul 2008 04:20:09 GMT

>Because of these defunct processes memory utilization is very high on server

Are you sure this memory use is due to the zombies? You may find there is another problem. You didn't use "kill -9" and have it work?

>Ya you are right wrapper is the zombie process.

Zombie master.

Bill: It would be a lot safer to reboot.

Then you might as well try killing the zombie master.

Re: Defunct processes

Rasheed Tamton — Tue, 22 Jul 2008 11:35:31 GMT

Hi,

I see the issue is discussed in very detail. Some times the below steps helps to find out what is going on with the zombie one

ps -el|grep defunct
truss -v all -cfp
lsof -p

Regards.

Re: Defunct processes

Dennis Handly — Tue, 22 Jul 2008 15:09:00 GMT

>Rasheed: Some times the below steps helps to find out what is going on with the zombie

tusc won't work on zombies, they are dead.
lsof -p is probably useless since the files opened by the zombie master wouldn't have much of a relation to the zombie.

Re: Defunct processes

Kavita Poonia — Tue, 22 Jul 2008 20:56:55 GMT

Hi Bill,

Thanks for you reply.

ebzwls01:/home # ipcs -bmop
IPC status from /dev/kmem as of Tue Jul 22 17:45:31 2008
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
m 0 0x411c0231 --rw-rw-rw- root root 0 348 788 13066
m 1 0x4e0c0002 --rw-rw-rw- root root 1 31040 788 13066
m 2 0x41202509 --rw-rw-rw- root root 1 8192 788 13066
m 4611 0x0c6629c9 --rw-r----- root root 2 19068672 1729 11959
m 4 0x06347849 --rw-rw-rw- root root 1 77384 1729 1829
m 1541 0x49182e06 --rw-r--r-- root root 0 22908 1723 1829
ebzwls01:/home # ps -ef|grep -i 1829
root 1829 1 0 Feb 10 ? 477:51 /opt/perf/bin/scopeux
ebzwls01:/home # ps -ef|grep -i 13066
root 13214 6494 0 17:54:20 pts/tb 0:00 grep -i 13066
ebzwls01:/home #

I have run ipcs -bmop command and the output pasted as above. There are two LPIDs for which NATTACH is 0. The LPID=1892 is running on server but LPID=13066 is not running.

Now what should I do ?

Re: Defunct processes

Kavita Poonia — Tue, 22 Jul 2008 21:04:30 GMT

Thanks Dennis for your reply.

Thanks Rasheed too........