Which process it using all my disk I/O ?

[David Love]

Hi Forum,

I have a problem on an HP-UX 11.11 server. Every so often disk I/O max's out at 100%. It will be like this for a few minutes then return to normal - it is apparently happening randomly and I have no idea what is causing it. I have tried to work it out by using Glance but to be honest, I'm not sure what I'm looking for.

Whilst disk I/O is at 100% how can I find out which process is causing it?

Many thanks,

Dave.

[Arunvijai_4 · Accepted solution]

Hi Dave,

You can use "sar" to find out. Do a "man sar" for more information.

-Arun

[Pete Randall]

Dave,

Do you use the GUI interface to Glance? If so, here is how I would approach this.

From the GPM Main window, select Reports > Process List. In the Process List window, select Configure > Sort Fields. That brings up a window where you highlight the field you want to sort on and move that field to the leftmost side - the field are sorted from left to right so moving Phys IO Rate all the way to the left will show you your most I/O intensive process.


Pete

[Alzhy]

If you are using Glance and you are seeing this 100% Disk IO utilisation AND you have not done any tweaks of you measureware package - then it is not an accurate porrayal of the actual IO performance. The Out of the Box Glance stats for Disk IO will flag the system as I/O bound even if just one disk is busy. Even the rationale for flagging a single disk as busy is suspect.

Use "sar -d" to see individual disk performance and observe if you have higher than normal average wait and service times. Pay close attention as well on which disks have queue lengths in excess of 0.

[Tim Nelson]

I concur with Pete.

If you have glance then sort the process screen by physical io/s. This should put the top user at the top of the list and now the process is identified.

[Sandman!]

Try to isolate the I/O bottleneck using sar and post the output here if possible:

# sar -d 5 50

cheers!

[Pete Randall]

Can one of you sar proponents explain to me how that is going to identify the process? I can see where it would identify the device, but Dave wants to know the process. Given this requirement, Glance would seem to be the obvious answer.


Pete

[Alzhy]

Peace and Clean Earth!

And to find out the top processes consuming IO bandwidth:

Save as top_io.ksh:

-----------------------------
#!/bin/ksh
#
trap 'rm tmp$$' 0 1 2 3
cat << EOF > /var/tmp/tmp$$

process loop {
print proc_proc_id,",",proc_parent_proc_id,",", proc_user_name,",",proc_proc_name,"," ,PROC_DISK_PHYS_IO_RATE
}
EOF

/opt/perf/bin/glance -j 2 -iterations 3 -adviser_only -syntax /var/tmp/tmp$$ 2>/dev/null

-----------------------

Then:
top_io.ksh|sort -t, -nr -k 5,5|head -10|awk -F, '{printf "%7.1f %5d %5d %8s %-30s\n", $5,$1,$2,$3,$4}'


Hope this helps..

[Tim D Fulford]

If you want to trace the **process(es)** responsile for the high IO, use glance (the ASCII version).

- select "o" for options
- edit the following line
Sort key (name/cpu/disk/rss): disk

and the top process is the one consuming the most IO. Jobs-a-good-un

If you want to trace the disk which is being hit the most sar is useful, ... but glance is better. (I am a fan of glance)

glance -u or use "u" option
This shows the disks utilisations of all the disks, and you can see which disk is being hit the most.

Regards

Tim