How many zombie processes is too many?

[Jan Shu]

Hi All,

I have zombie (oracle) processes on 2 production HP9000 N4000 database servers. The number of defunct processes increasing 5 - 7 per day. So far they (50 defunct) do not take up much CPU or memory resource yet. What is the maximum number of zombies that I should consider to reboot the servers?

Thank you for your help in advance.

Best regards,
Jan Shu

[A. Clay Stephenson · Accepted solution]

You are really asking the wrong question. You need to be asking why you are getting zombie processes at all and that is what you need to attack. Most likely PC client are not disconnecting or termiinating sessions correctly -- a little training is in order or possibly physical adjustments of sloppy programmers. I've seen systems run just fine with hundreds of zombies in the process table -- most do no real harm other than annoying sysadmins but they are telling you
that sloppy practices are in play.

[Steven E. Protter]

1 is too many.

If you can't get them will a kill -9 you have to boot to fix it.

The fact that your system is breeding zombies is bad and can hurt peformance a lot. Check the owner of the processes and contact the application vendor. You need to do something to the app to make this stop.

SEP

[RAC_1]

I have never came across a situation when there were so many zombies. And I think there no such no. as so many zombies should not be there.

The real cause of worry should be why those many zombies?? What are they waiting for?? Those are in kernel mode and are waiting for I/O to get completed.

Anil

[Scot Bean]

I would want to find out why the zombies are happening. Generally not healthy, means some processes are not running properly.

[Chris Wilshaw]

I've seen this kind of zombie spawning before - as Clay suggested, it could be that users are not logging out properly, from their sessions.

Last year, we found one of our systems spawning zombies at a rate of 2-3 per hour. It turned out that a trap statement had been added to one of the scripts, that prevented the processes from dying when the users logged out by just switching off their terminals.

The trap statement was

trap "" 1 2 3

This was amended to

trap "" 2 3

which resolved the problem. signal 1 is the SIGHUP signal (use kill -l to list the signals)

[Bill Hassell]

To expand on Chris' trap problem, this has happened to a lot of admins who looked at the two trap statements in /etc/profile and figured the second one was a typo, so it was removed. trap 1 2 3 and trap "" 1 2 3 are *VERY* different. The first changes signals 1 2 and 3 to default behavior but the second ignores the 3 signals. This is desirable during the execution of /etc/profile but traps must re-enabled before /etc/profile ends or you'll all sorts of nasty processes (not just zombies) in the mix.

A zombie is a dead program, which is why they are called zombies and cannot be killed with kill -9 (they're already dead). Since there is no code in memor and nothing is executing, the only resource is the process table, sized by nproc in the kernel. If you get thousands, there will be a point where not even root can login because no more processes can be run (no room in the process table). AS mentioned, zombies are an indication of unintentional oversights (OK, bad programming) an never happen on a well managed system. Since they are Oracle processes, the first step is to consult the Oracle knowledge base and the second is probably to apply the latest patches.

[Jan Sladky]

easy to say,

there shouldn't be any zombie, well administrated system should be free of zombie because they can consume lot of CPU or memory and thus decrease server performance.

So try figure out the cause of the zombies, to clear them try kill, kill -9 or if it is possible ( be carefull! ) kill PPID

br Jan

[Ahmed_42]

Let me clarify Jan's email. We know the reason of the zombies and we know how to fix it but we can't apply the fix. The zombies were generated because of third party software that we had to install in production to get some client work done.

The vendor is working on a patch to fix the problem but we have to live with these zombies for a sometime. That is why Jan is asking about the number of zombies that should trigger a red light and lead to a reboot.

Any ideas?

[James Lynch]

Ahmed,

Bill's description is right on the money. Zombie processes are the result of the parent process not cleaning up after it's child process has died. In order for this cleanup to occur, the Parent process must issue a "wait" system call. The "wait" system call will return the staus of the child process, usually the return or exit code of the dead child. Until the parent process issues the "wait" or the parent dies, the kernel will maintin the dead child's status in the form of a zombie process. THE ZOMBIE PROCESS DOES NOT CONSUME ANY MEMORY OR CPU RESOURCES, it is just using one entry in the nproc table.

If the parent process does not care about the status of its children (dead or alive) then that parent process should ignore the SIGCHLD signal. Doing so tells the kernel that the parent process does not care about the death of its children and the kernel will not save the process status when any of its children die, thus preventing the zombie proccesses from being created.

Now, there are anly three ways to get rid of zombie processes.

1) The parent must issue the "wait" system call. We already know that this is not going to happen, so that's out. At least not until the third party vendor fixes their code.

2) The parent process of the zombies dies. When a parent process dies (even zombies have a parent) all of its children,including the zombies, get owned by the "init" process. The "init" process will automatically isue the "wait" system call for each of the orphaned zombie processes that it inheirents. So, killing the zombie's parent process would cause the zombies to get removed. This may or may not be an option.

3) You reboot the sytem.

Hope that helps.

-JL

[A. Clay Stephenson]

In this case, since you are getting the problem fixed (although this is a serious bug to have slipped through any developer's quality control), I would setup a cronjob that runs every 30 minutes or so that does a sar -v. You then see how close to nproc the current process count is. It's probably time to schedule a reboot when the proc table approaches 75% or so. You can also buy yourself more time by bumping up nproc but that is really throwing a Band-Aid at the problem.

I would really press the software vendor to explain the problem. If the explanations seem less than clear, that indicates that the developers don't have a firm grasp of the underlying problem.

[Jan Shu]

Hi Clay,

Thanks for your help. I will set up this cron to monitoring nproc.

Have a great weekend.

Best Regards,
Jan