how to tell which processes are causing i/o bottleneck

[Johan Hoeke]

Using top, we see that our RHEL 3 taroon update 3 server DL380 G3 server shows a high % of iowait. iostat -x reveals that it's on /dev/cciss/c0d0p2, that is used by logical volume /dev/Vol01/LV01_opt. %util show 190% (!!) in the sample below. So we have narrowed
it down to this disk. But how can we tell what processes are hammering this
disk?

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
/dev/cciss/c0d0p2
0.00 0.00 0.00 686.30 0.00 5459.36 0.00 2729.68 7.95
178.58 26.30 2.78 190.87

[Rick Garland]

Under the "System Tools" menu item, you will find a "System Monitor" that will display PIDs as well as the corresponding name of the process.

Note, this is an installable app.

If installed you can find in /usr/bin/gnome-system-monitor

[xyko_1]

Hi,

if you don't determine any application at all, verify if you are not running out of memory. The high intence io my be paging/swap activity.
Does /dev/cciss/c0d0p2 is your swap partition or does you have some extra swapfile defined there ?

Just a guess.

Regards,
Xyko

[Johan Hoeke]

thanks for the answers so far.

nope, no swap or anything like that there.
the processes there are cloverleaf / quovadx sites. (communication engine for enterprise application integration like sybase impact or microsoft bizztalk). 15 or so sites with their own processes. can't tell which is causing the high i/o. looking for a top, but then sorted by i/o or something similar.

[xyko_1]

Well,

one thing that may help you is to determine what processes have opened files on the affected file system.

lsof /dev/Vol01/LV01_opt
or
lsof /dev/cciss/c0d0p2

will show you processes and opened files.

The second step is to guess who is the best wish to be the guilty. One good choice is the top cpu consuming process, that you can determine using top.

Regards,
Xyko

[dirk dierickx]

as far as i know, the only tool i know that can display his kind of information is HP Glance.

i'm pretty sure it is possible to write this yourself in C. otherwise you have to play the guessing game with hit and miss.

but then again, if you have so much IO, i would think the process is also using a lot of CPU time.

[Johan Hoeke]

thanks for the replies again guys.

Rick: tried gnome-system monitor. fancy top, like kde system guard. But I found no i/o specific stuff.

xyko: i know lsof, but thanks.

Dirk: am running a trial of glance for linux. it is telling me i've got a disk bottleneck. but I can't get it to show me what process is consuming a lot of i/o

[Johannes Krackowizer_1]

hi Johan,

i'm sorry, i don't know any tool to see wich process do the havy load. But my advise is to stop one process after the other and have look on the harddisk load after stoping a process to see which process did the havy load.

johannes

[Rick Beldin]

Rather than stopping processes, you could use lsof to see which process has open fd's on the disk/partition in question. You could then use strace to trace the processes while they are running to see which ones are doing a lot of read/write/stat/access calls. Probably have to do something like:

strace pid | grep -E 'read|write'

Sampling would be the best way, so some profiling tool might be useful. There are a couple of projects such as:

http://oss.sgi.com/projects/kernprof/

which may be a start.

[xyko_1]

Hi Johan,

doing some research on the net I found acct.

http://www.gnu.org/software/acct/

Try to accton for a while, when the high i/o occurs, and see if the log helps you.

But don't let the system account on forever. It may consume a lot of computer resource and slowdown your server.

Regards,
Xyko

[Johan Hoeke]

thanks for the info everybody. no obvious answer. work to be done.

[Johan Hoeke]

thanks for the info everybody. no obvious answer. work to be done.