topic Re: how to tell which processes are causing i/o bottleneck in Operating System - Linux

how to tell which processes are causing i/o bottleneck

Johan Hoeke — Thu, 07 Oct 2004 09:11:24 GMT

Using top, we see that our RHEL 3 taroon update 3 server DL380 G3 server shows a high % of iowait. iostat -x reveals that it's on /dev/cciss/c0d0p2, that is used by logical volume /dev/Vol01/LV01_opt. %util show 190% (!!) in the sample below. So we have narrowed
it down to this disk. But how can we tell what processes are hammering this
disk?

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
/dev/cciss/c0d0p2
0.00 0.00 0.00 686.30 0.00 5459.36 0.00 2729.68 7.95
178.58 26.30 2.78 190.87

Re: how to tell which processes are causing i/o bottleneck

Rick Garland — Thu, 07 Oct 2004 10:20:41 GMT

Under the "System Tools" menu item, you will find a "System Monitor" that will display PIDs as well as the corresponding name of the process.

Note, this is an installable app.

If installed you can find in /usr/bin/gnome-system-monitor

Re: how to tell which processes are causing i/o bottleneck

xyko_1 — Thu, 07 Oct 2004 12:54:21 GMT

Hi,

if you don't determine any application at all, verify if you are not running out of memory. The high intence io my be paging/swap activity.
Does /dev/cciss/c0d0p2 is your swap partition or does you have some extra swapfile defined there ?

Just a guess.

Regards,
Xyko

Re: how to tell which processes are causing i/o bottleneck

Johan Hoeke — Thu, 07 Oct 2004 13:06:14 GMT

thanks for the answers so far.

nope, no swap or anything like that there.
the processes there are cloverleaf / quovadx sites. (communication engine for enterprise application integration like sybase impact or microsoft bizztalk). 15 or so sites with their own processes. can't tell which is causing the high i/o. looking for a top, but then sorted by i/o or something similar.

Re: how to tell which processes are causing i/o bottleneck

xyko_1 — Thu, 07 Oct 2004 14:34:31 GMT

Well,

one thing that may help you is to determine what processes have opened files on the affected file system.

lsof /dev/Vol01/LV01_opt
or
lsof /dev/cciss/c0d0p2

will show you processes and opened files.

The second step is to guess who is the best wish to be the guilty. One good choice is the top cpu consuming process, that you can determine using top.

Regards,
Xyko

Re: how to tell which processes are causing i/o bottleneck

dirk dierickx — Fri, 08 Oct 2004 01:36:43 GMT

as far as i know, the only tool i know that can display his kind of information is HP Glance.

i'm pretty sure it is possible to write this yourself in C. otherwise you have to play the guessing game with hit and miss.

but then again, if you have so much IO, i would think the process is also using a lot of CPU time.

Re: how to tell which processes are causing i/o bottleneck

Johan Hoeke — Fri, 08 Oct 2004 01:51:59 GMT

thanks for the replies again guys.

Rick: tried gnome-system monitor. fancy top, like kde system guard. But I found no i/o specific stuff.

xyko: i know lsof, but thanks.

Dirk: am running a trial of glance for linux. it is telling me i've got a disk bottleneck. but I can't get it to show me what process is consuming a lot of i/o

Re: how to tell which processes are causing i/o bottleneck

Johannes Krackowizer_1 — Fri, 08 Oct 2004 03:56:05 GMT

hi Johan,

i'm sorry, i don't know any tool to see wich process do the havy load. But my advise is to stop one process after the other and have look on the harddisk load after stoping a process to see which process did the havy load.

johannes

Re: how to tell which processes are causing i/o bottleneck

Rick Beldin — Fri, 08 Oct 2004 06:44:03 GMT

Rather than stopping processes, you could use lsof to see which process has open fd's on the disk/partition in question. You could then use strace to trace the processes while they are running to see which ones are doing a lot of read/write/stat/access calls. Probably have to do something like:

strace pid | grep -E 'read|write'

Sampling would be the best way, so some profiling tool might be useful. There are a couple of projects such as:

http://oss.sgi.com/projects/kernprof/

which may be a start.

Re: how to tell which processes are causing i/o bottleneck

xyko_1 — Fri, 08 Oct 2004 07:50:05 GMT

Hi Johan,

doing some research on the net I found acct.

http://www.gnu.org/software/acct/

Try to accton for a while, when the high i/o occurs, and see if the log helps you.

But don't let the system account on forever. It may consume a lot of computer resource and slowdown your server.

Regards,
Xyko

Re: how to tell which processes are causing i/o bottleneck

Johan Hoeke — Fri, 08 Oct 2004 09:22:38 GMT

thanks for the info everybody. no obvious answer. work to be done.

Re: how to tell which processes are causing i/o bottleneck

Johan Hoeke — Fri, 08 Oct 2004 09:23:25 GMT

thanks for the info everybody. no obvious answer. work to be done.