topic Re: Recommendations: Defunct Processes in Operating System - HP-UX

Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 13:32:21 GMT

We have a scripts that go out and check for defunct processes. What should we do once we find them???? We typically have 3 or 4 servers with defunct processes but we don't know what action we should take with them.

Thanks

Scott

Re: Recommendations: Defunct Processes

Sanjay_6 — Tue, 11 Jan 2005 13:36:42 GMT

Hi Scott,

Do a "ps -ef" and grep for defunt. Try a "kill -9 PID" to kill the defunt processes. you may be able to kill them or you may not. If you can't kill them, a reboot is the only way to clear them.

Hope this helps.

Regds

Re: Recommendations: Defunct Processes

Pete Randall — Tue, 11 Jan 2005 13:37:50 GMT

Scott,

First you should beat your programmers soundly for leaving such sloppy code around.

Second, you can try to kill them off. Start with kill -15, then kill -11 and only resort to kill -9 out of desparation. Sometimes, the processes won't die. In that case, you can sometimes get rid of them by killing the parent. However, if they're a zombie, their parent process ID is now 1 - init - and you don't want to kill that.

You really should be spending more time on why you're getting defunct processes than you are on cleaning them up.

Pete

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 13:40:25 GMT

Pete,

I understand, but when you are at the mercy of a major application vendor specific for hospital applications, beating them (although I would not be against that) really isn't an option.

Re: Recommendations: Defunct Processes

John McWilliams_1 — Tue, 11 Jan 2005 14:04:42 GMT

Hi Scott

I always try kill -15 on the PID. This normally kills them. If that fails I use kill -9 with a 99% success rate.

Cheers John

Re: Recommendations: Defunct Processes

David DeWitt_2 — Tue, 11 Jan 2005 14:17:13 GMT

Scott,

If you're going to automate killing them (I wouldn't) you could probably add a log file to record things such as date/time, PID, Process Name. That may help both if the kills start causing other problems and as evidence to the vendor.

Has the vendor admitted that their app creates defunct processes?

-dave

Re: Recommendations: Defunct Processes

Pete Randall — Tue, 11 Jan 2005 14:19:32 GMT

Scott,

OK, if beating them is only a fond dream, how about regular re-boots? Do your uptime requirements allow for a nightly/weekly/fortnightly window where you could re-boot and clean up the processes that way? It would be simple enough to cron a reboot job.

Pete

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 14:19:52 GMT

David,

We are not looking to automate the killing, just curious about what our next steps might be. I like you idea of a log file. That would be nice.

Scott

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 14:21:51 GMT

Pete,

Reboots are not an option. We are fighting now trying to get a 2 hour maintenance window once a quarter. Regular reboots would not fly. I guess being a hospital makes a difference. Our apps have to be up 24/7 for patient care.

Scott

Re: Recommendations: Defunct Processes

Pete Randall — Tue, 11 Jan 2005 14:26:00 GMT

I didn't really think that would work for you but had to suggest it.

Sorry,

Pete

Re: Recommendations: Defunct Processes

David DeWitt_2 — Tue, 11 Jan 2005 14:28:11 GMT

Scott,

Do you have anyone capable of writing a script to recognize new defuncs and time-tag them in a log file? I've done similar scripts in the past and may be able to find them if you need a sample.

With a little work you could probably receive an e-mail anytime a new defunc shows up.

-dave

Re: Recommendations: Defunct Processes

Vitek Pepas — Tue, 11 Jan 2005 14:29:31 GMT

Correct me if I'm wrong, but isn't defunct just an entry in the process table and not a running process? Killing defuncts makes no sense then, only killing their parent process may help to get rid of them.

Re: Recommendations: Defunct Processes

A. Clay Stephenson — Tue, 11 Jan 2005 14:57:18 GMT

Vitek is quite correct; although defunct processes may be driving you (the admin) crazy, they only occupy a slot in the process table and consume no other resources beyond that. Sending a signal to them (kill) is utterly pointless. As others have indicated, the real solution is to prevent them in the first place. Often, user education is all that is needed. Instructing them in proper application exit procedures may greatly reduce your number of zombies. You should also make your application vendor aware of the issue; I always want to know when anything I've written has problems.

What you really need to be on the lookout for is runaway processes (which can also result from improper termination/exit procedures) because unlike zombies, runaways can cripple your box.

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 15:05:42 GMT

We have a script that looks at the output of top to determine runaway processes. Is there any point in monitoring defunct processes based on Clay's response???

Re: Recommendations: Defunct Processes

Vitek Pepas — Tue, 11 Jan 2005 15:26:40 GMT

You could monitor the number of defuncts. Large number may have impact on performance, due to extensive process table searches.
The limit of processes per user may also be exceeded because of defuncts.

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 15:28:34 GMT

On any given day, we only see one defunct process. We check this every 15 minutes (when we do our other system checks).

Re: Recommendations: Defunct Processes

Vitek Pepas — Tue, 11 Jan 2005 15:40:38 GMT

In this case you have nothing to worry about.

I work for one of the major hospital sofware companies myself - my advise: go get'em!

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 15:42:18 GMT

Vitek,

Hopefully, I didn't offend you!!

Thanks for the advise.

Re: Recommendations: Defunct Processes

Scott Frye_1 — Tue, 11 Jan 2005 15:43:31 GMT

Based on Clay and Vitek's responses, I don't believe we have anything to worry about. More than likely we will just remove that piece of our monitoring script so we won't get notified every 15 minutes.

Thanks!!!!

Scott