Recommendations: Defunct Processes

[Scott Frye_1]

We have a scripts that go out and check for defunct processes. What should we do once we find them???? We typically have 3 or 4 servers with defunct processes but we don't know what action we should take with them.

Thanks

Scott

[Sanjay_6]

Hi Scott,

Do a "ps -ef" and grep for defunt. Try a "kill -9 PID" to kill the defunt processes. you may be able to kill them or you may not. If you can't kill them, a reboot is the only way to clear them.

Hope this helps.

Regds

[Pete Randall]

Scott,

First you should beat your programmers soundly for leaving such sloppy code around.

Second, you can try to kill them off. Start with kill -15, then kill -11 and only resort to kill -9 out of desparation. Sometimes, the processes won't die. In that case, you can sometimes get rid of them by killing the parent. However, if they're a zombie, their parent process ID is now 1 - init - and you don't want to kill that.

You really should be spending more time on why you're getting defunct processes than you are on cleaning them up.


Pete

[Scott Frye_1]

Pete,

I understand, but when you are at the mercy of a major application vendor specific for hospital applications, beating them (although I would not be against that) really isn't an option.

[John McWilliams_1]

Hi Scott

I always try kill -15 on the PID. This normally kills them. If that fails I use kill -9 with a 99% success rate.

Cheers John

[David DeWitt_2]

Scott,

If you're going to automate killing them (I wouldn't) you could probably add a log file to record things such as date/time, PID, Process Name. That may help both if the kills start causing other problems and as evidence to the vendor.

Has the vendor admitted that their app creates defunct processes?

-dave

[Pete Randall]

Scott,

OK, if beating them is only a fond dream, how about regular re-boots? Do your uptime requirements allow for a nightly/weekly/fortnightly window where you could re-boot and clean up the processes that way? It would be simple enough to cron a reboot job.


Pete

[Scott Frye_1]

David,

We are not looking to automate the killing, just curious about what our next steps might be. I like you idea of a log file. That would be nice.

Scott

[Scott Frye_1]

Pete,

Reboots are not an option. We are fighting now trying to get a 2 hour maintenance window once a quarter. Regular reboots would not fly. I guess being a hospital makes a difference. Our apps have to be up 24/7 for patient care.

Scott

[Pete Randall]

I didn't really think that would work for you but had to suggest it.

Sorry,

Pete

[David DeWitt_2]

Scott,

Do you have anyone capable of writing a script to recognize new defuncs and time-tag them in a log file? I've done similar scripts in the past and may be able to find them if you need a sample.

With a little work you could probably receive an e-mail anytime a new defunc shows up.

-dave

[Vitek Pepas]

Correct me if I'm wrong, but isn't defunct just an entry in the process table and not a running process? Killing defuncts makes no sense then, only killing their parent process may help to get rid of them.

[A. Clay Stephenson]

Vitek is quite correct; although defunct processes may be driving you (the admin) crazy, they only occupy a slot in the process table and consume no other resources beyond that. Sending a signal to them (kill) is utterly pointless. As others have indicated, the real solution is to prevent them in the first place. Often, user education is all that is needed. Instructing them in proper application exit procedures may greatly reduce your number of zombies. You should also make your application vendor aware of the issue; I always want to know when anything I've written has problems.

What you really need to be on the lookout for is runaway processes (which can also result from improper termination/exit procedures) because unlike zombies, runaways can cripple your box.

[Scott Frye_1]

We have a script that looks at the output of top to determine runaway processes. Is there any point in monitoring defunct processes based on Clay's response???

[Vitek Pepas]

You could monitor the number of defuncts. Large number may have impact on performance, due to extensive process table searches.
The limit of processes per user may also be exceeded because of defuncts.

[Scott Frye_1]

On any given day, we only see one defunct process. We check this every 15 minutes (when we do our other system checks).

[Vitek Pepas · Accepted solution]

In this case you have nothing to worry about.

I work for one of the major hospital sofware companies myself - my advise: go get'em!

[Scott Frye_1]

Vitek,

Hopefully, I didn't offend you!!

Thanks for the advise.

[Scott Frye_1]

Based on Clay and Vitek's responses, I don't believe we have anything to worry about. More than likely we will just remove that piece of our monitoring script so we won't get notified every 15 minutes.

Thanks!!!!

Scott