Tool needed for associating HD I/O load to processes

[Ralph Grothe]

Hello everyone,

I have a system with massive performance bottlenecks as far as hard disk I/O is concerned.
We already had the disk array subsystem and cache checked by the vendors' experts to no avail.
Because I want to trace the culprit I am in search of a tool that can identify those processes which are responsible for the I/O blocking.
We have PerfView to draw fancy charts though, but from those you cannot identify the processes.
I haven't yet found the time to get familiar with the adviser's syntax to trigger my own alarms etc.
Maybe PV is already the right tool.
I also thought of something like lsof but couldn't compile it with gcc on this 64bit machine, and the precompiled binary you can download from the Ports Collection is also 32bit (wonder why swinstall installed it anyway).
Maybe anyone knows of other freeware tools, or has hints what and how to trace?

Regards
Ralph

[MANOJ SRIVASTAVA]

Hi Ralph

there are many ways to do it :

1. Native Unix tools is SAR

do a sar -d 2 10 and it will give you the avergaes too so there is simple way of looking at it.

2. Glance would give you a nice o/p interms of top 10 disk being thrashed .

3. In PV you can set the procees / user names which u want to track so that it keep data for the application and you can see it .

4. Download from www.sarcheck.com a simple free trial version of sarcheck which gives a very good analysis of the whole system using sar/ ps outputs .

5.In case you can use lsof ( 64 bit) version (compiled version which works )is avialable at :

ftp://vic.cc.purdue.edu/pub/tools/unix/lsof

and incase you are not able to comile send me your email id I will email the compiled version



Manoj Srivastava

[Bill McNAMARA_1]

http://www.software.hp.com/products/performance/perfview.html

is an interesting complete performance analysis tool.

For what you want Glance is the closest thing you'll get.

So other usefull sysadmin s/w can be found here:
http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/alpha.html
Later,
Bill

[Wodisch]

Hello Ralph,

what you need could be "MWA" (MeasureWareAgent)
nowadays called "OpenView Performance Agent"...
Check the manual pages for this tool from the OpenView
manual page server:
http://ovweb.external.hp.com/lpe/doc_serv/

HTH,
Wodisch

[Patrick Wallek]

Since you are using PerfView, you already have MeasureWare installed.

PV will allow you to drill down into the data to see which process are running at any point in time.

If you bring up the disk chart in PV and highlight an area of time, you can then drill down and see what disks are being used and I think you can also see which processes are hitting the disks. Read up on PV. It should be able to get to the data that you want. Keep in mind that PV / MeasureWare is data from the past.

If you want to see instantaneous data, Glance is the tool to use. There is a trial available on one of the application CDs.

You could also use top to look at some things, though you won't get near the data you will from Glance.

[A. Clay Stephenson]

Hi Ralph,

As has been mentioned, Glance is the perfect tool for this. Although you didn't mention what applications are running, my psychic connections lead me to believe that this is a database server. Problems like this are very common in those environments and is almost always due to poorly written code. I have seen many cases where all the OS and hardware tuning in the world improved things only 3-4% and yet adding one index improved performance 10x +.

When you get Glance installed or some Perfview drill-downs done, if you will post those, I'm sure someone can offer advice.

Regards, Clay

[A. Clay Stephenson]

Hi again Ralph,

I had another thought. Are you really having performance problems or are you misreading Perfview/MeasureWare data? Since you are using a disk array, remember that your UNIX box sees only 1 (or a few devices) which may in fact be 10's or even 100's of physical disks. The host doesn't have a clue about this and only sees that a huge amount of I/O is going through a few (possibly 1) devices. Often SA's will divide an array into several LUN's so that performance APPEARS better when actually performance had not changed significantly.

Food for thought, Clay

[Michael Tully]

Hi Ralph,

There is another tool that can be used to monitor system processing and that is 'tusc'. It is very simple to use and gives good output for analysis.

The good news is that it is free!
The bad news is that there is no source code and only available on HP-UX 11.

ftp://ftp.cup.hp.com/dist/networking/misc/

HTH
Michael