Killing a process waiting on I/O

[Ronelle van Niekerk]

This may be a bit vague, but here we go:

We have a job that runs on the weekend that sometimes needs to kill user processes that are accessing the same database files.

Unfortunately sometimes it cannot kill (even with a -9) these processes. I think this may be because the processes are waiting on I/O and can't be killed until they recieve their answer.

My question is:
How does the waiting on I/O work, is there any chance to kill the process?

-Ronelle

[Jeff Schussele]

Hi Ronelle,

Unfortunately, one can never kill a process waiting on I/O because it's priority is in the kernel range & these are unkillable in that state. Only way to kill these would be to kill the parent & hope it reaps it's child.

Rgds,
Jeff

[Bill Hassell]

Very common problem. The program is trying to access I/O without any timeout and basic Unix design says that kill -9 sets a signal in the process table (SIGKILL) and the process will terminate when it begins to run. If the I/O never completes, the process will never run. A reboot is about the only way to get rid of these programs (and rewrite them so they have a timeout). Another possibility is to try to create a core dump--this sometimes kills the process:

ulimit -c 0
kill -3

where is the process ID you are trying to kill.

[A. Clay Stephenson]

You have to understand what a kill (signal), even a kill -9, is; it justs sets a flag in the process table. The next time that the process is running it will be handled BUT if a process is blocked on I/O it ain't running so the signal is never caught and acted upon. Of all the UNIX system calls, kill() is perhaps the most poorly named. At best, it's a suggestion.

What you can do, for example, if waiting on a tape drive, is to eject the tape or power off the device in question. Then the process is no longer blocked on I/O and can respond to your signal.

By the way, a kill -9 should be your weapon of last resort because absolutely no cleanup (removal of temp files; detach shared memory; etc. is or can be done).

[Ronelle van Niekerk]

What you guys seem to be describing are processes that get stuck waiting on I/O.
The processes the users are running - the ones than need to be killed - are not stuck, they do get killed eventually or end by themselves. I know this because they're not there the next morning but I'm not sure if they die because the kill -9 flag was set against them or if they ended properly.

So it works as such:
1) Our program finds a user process is reading a file it needs
2) it tries to kill the process
3) if the process is still running it tries to kill -9 the process
4) if the process is still running then our program aborts.

I'm not sure how to ask what I need to know:
What exactly does "waiting on I/O" mean?
Is the user process WAITING to receive data or is the user process BUSY receiving data?
Is there a "window" during the waiting on I/O that the process will run - and therefore die? Or how long would it be waiting on I/O?

Would it help if we changed the program to:
1) Our program finds a user process is reading a file it needs
2) it tries to kill the process
3) if the process is still running it tries to kill -9 the process
4) if the process is still running then our program waits a while (10 minutes?)
5) if the process is still running then our program aborts.

Sorry to be so long-winde

[James Murtagh · Accepted solution]

Hi Ronelle,

If a process is waiting on IO then it has performed a system call and is in kernel mode but in process context. The process is waiting on the driver to return the data or the result of the data transfer from the physical device - in this state it is said to be blocked - it goes to sleep waiting on the IO to complete, allowing other processes to run. To prevent corruption of the kernel the process is assigned a sleep priority in this state, which will be in the externel priority range 128-153, termed system priority and uninteruptable. See /usr/include/sys/param.h for a list of priorities relating to different types of IO.

A process will receive signals in one of a few conditions, on return to user mode from a system call is one of them. Also, certain signals can be caught, a few, like SIGKILL cannot. Hence your program will have been killed by this signal after returning to user mode.

To prevent processes waiting on IO that cannot be performed the driver normally sets a timeout period where it will return the failure to the requestor. For example take the pvtimeout on lvm disk devices. This allows lvm to switch to a differnt path if pvlinks are configured. However, if the data cannot be accessed from the other link eventually the lvtimeout will be reached and the failure sent back to the application, which then has the responsibility to handle it. If you are not receiving a timeout or see the process waiting on IO for a lengthy period of time I assume there is some sort of performance problem on the device you are accessing.

I think the best course of action is if you determine what IO the processes are waiting on and fix the underlying problem - running a script to kill the process after it returns from the IO request probably doesn't make much sense. If you post the sleep priority we should be able to figure this out.

Sorry to be so long winded too, its not an easy question.

Cheers,

James.

[Ronelle van Niekerk]

Thanks James,
that does help, I'll investigate further.

-Ronelle