topic Re: Which process it using all my disk I/O ? in Operating System - HP-UX

Which process it using all my disk I/O ?

David Love — Tue, 21 Mar 2006 08:20:53 GMT

Hi Forum,

I have a problem on an HP-UX 11.11 server. Every so often disk I/O max's out at 100%. It will be like this for a few minutes then return to normal - it is apparently happening randomly and I have no idea what is causing it. I have tried to work it out by using Glance but to be honest, I'm not sure what I'm looking for.

Whilst disk I/O is at 100% how can I find out which process is causing it?

Many thanks,

Dave.

Re: Which process it using all my disk I/O ?

Arunvijai_4 — Tue, 21 Mar 2006 08:24:18 GMT

Hi Dave,

You can use "sar" to find out. Do a "man sar" for more information.

-Arun

Re: Which process it using all my disk I/O ?

Pete Randall — Tue, 21 Mar 2006 08:26:53 GMT

Dave,

Do you use the GUI interface to Glance? If so, here is how I would approach this.

From the GPM Main window, select Reports > Process List. In the Process List window, select Configure > Sort Fields. That brings up a window where you highlight the field you want to sort on and move that field to the leftmost side - the field are sorted from left to right so moving Phys IO Rate all the way to the left will show you your most I/O intensive process.

Pete

Re: Which process it using all my disk I/O ?

Alzhy — Tue, 21 Mar 2006 09:29:32 GMT

If you are using Glance and you are seeing this 100% Disk IO utilisation AND you have not done any tweaks of you measureware package - then it is not an accurate porrayal of the actual IO performance. The Out of the Box Glance stats for Disk IO will flag the system as I/O bound even if just one disk is busy. Even the rationale for flagging a single disk as busy is suspect.

Use "sar -d" to see individual disk performance and observe if you have higher than normal average wait and service times. Pay close attention as well on which disks have queue lengths in excess of 0.

Re: Which process it using all my disk I/O ?

Tim Nelson — Tue, 21 Mar 2006 12:59:13 GMT

I concur with Pete.

If you have glance then sort the process screen by physical io/s. This should put the top user at the top of the list and now the process is identified.

Re: Which process it using all my disk I/O ?

Sandman! — Tue, 21 Mar 2006 13:12:24 GMT

Try to isolate the I/O bottleneck using sar and post the output here if possible:

# sar -d 5 50

cheers!

Re: Which process it using all my disk I/O ?

Pete Randall — Tue, 21 Mar 2006 13:26:02 GMT

Can one of you sar proponents explain to me how that is going to identify the process? I can see where it would identify the device, but Dave wants to know the process. Given this requirement, Glance would seem to be the obvious answer.

Pete

Re: Which process it using all my disk I/O ?

Alzhy — Tue, 21 Mar 2006 13:36:00 GMT

Peace and Clean Earth!

And to find out the top processes consuming IO bandwidth:

Save as top_io.ksh:

-----------------------------
#!/bin/ksh
#
trap 'rm tmp$$' 0 1 2 3
cat << EOF > /var/tmp/tmp$$

process loop {
print proc_proc_id,",",proc_parent_proc_id,",", proc_user_name,",",proc_proc_name,"," ,PROC_DISK_PHYS_IO_RATE
}
EOF

/opt/perf/bin/glance -j 2 -iterations 3 -adviser_only -syntax /var/tmp/tmp$$ 2>/dev/null

-----------------------

Then:
top_io.ksh|sort -t, -nr -k 5,5|head -10|awk -F, '{printf "%7.1f %5d %5d %8s %-30s\n", $5,$1,$2,$3,$4}'

Hope this helps..

Re: Which process it using all my disk I/O ?

Tim D Fulford — Tue, 21 Mar 2006 16:04:33 GMT

If you want to trace the **process(es)** responsile for the high IO, use glance (the ASCII version).

- select "o" for options
- edit the following line
Sort key (name/cpu/disk/rss): disk

and the top process is the one consuming the most IO. Jobs-a-good-un

If you want to trace the disk which is being hit the most sar is useful, ... but glance is better. (I am a fan of glance)

glance -u or use "u" option
This shows the disks utilisations of all the disks, and you can see which disk is being hit the most.

Regards

Tim