How can I measure IO of disks

[DavidJ]

Can anyone tell me what I can use to measure throughput to a disk subsystem?
We are experimenting with different configs and setups and our DB shows it's spending a lot of time waiting on IO.
scsi_max_qdepth 128
OS 11i and 11.23

T.I.A
David

[Wouter Jagers]

Hi,

Glance is a great tool for these things, should you have it (it costs money).

Otherwise, sar could be your friend.. check out its -d option.

Cheers,
Wout

[DavidJ]

Ta Wout,

But I SHOULD have mentioned that I am using SAR and GLANCE but I am looking at something more substantial, not quite sure what that is yet ;-)

[Wouter Jagers]

Hehe, oh I see :-)

Have you checked out the HP OpenView Performance Agent (OVPA) ?

No first hand experience with it myself, but I've seen MeasureWare before and that worked nicely.

Cheers !

[Steven E. Protter]

Shalom,

http://www.hpux.ws/system.perf.sh

Scripted peformance patch set that provides a little more sophitication.

As to the problem at hand:
1) Check the disk subsystems for proper configuration. This often slows performance.
2) Have the app developers make sure their sql code is efficient
3) See that LVM and SCSI patches as well as the latest bi-annual patch set are installed on the system.
4) Run oracle stats pack and see what queries are using most of the system i/o.

SEP

[DavidJ]

Wout,

OVPA? I'd rather extract my teeth without anaesthetic ;-( But then that's MY experience..

SEP

1) Done and confirmed

2)see 1 above

3)see 1 & 2 above

4) using Sybase but yes we are looking there using a few DB tools.

Will look at the script.

Thanks

[Wouter Jagers]

Haha, ok.. thanks for the warning :-)

Cheers

[Steve Lewis]

What you are looking at is possibly not so much a problem in itself but a symptom.

It all starts and ends with the application. You need to understand why it is going to the DB for data in the first place.

Then look at the DB and understand why it is going to the disk array. You need to find out which tables are the hardest-hit and if they are being scanned, heavily updated, if its the logging that is slow, the stats, or not enough DB buffercache memory.

You have looked at the disk stats and adjusted your queue depth (128 is possibly too deep IMHO, remember the queue depth is per lun, not overall). Also queue depth adjustment will only push more work directly into the array. If one particular lun is heavily used, then you must spread the load over several luns on different spindles, then look into multiple or faster controllers, or more memory in the array cache and so on.

If you have a SAN switch or 2, then those give throughput figures as well.

Look at the array stats itself (Commandview, armlog, armdsp,armdsp) and try to understand why you are getting large delays and which luns they are, then how those luns are configured. If your array is overloaded then its time to look at a second array, a bigger array, or having more disks.
Look at the tpc.org systems design and you will find all the DB and server manufacturers specifying thousands of small, fast disks, often in a large number of disk arrays, in preference to a single big array with a few big disks. The performance difference is huge, even these days.

[DavidJ]

Steve,

>>It all starts and ends with the application.<< That is what we all would like to believe ;-) but unfortunately "WE" ALWAYS have to prove that there is NO problem on the system BEFORE the apps "people", if we can call them that, will look at the app :-)

The disk array is an EVA8k and that looks like it's sleeping, cache is not the problem as we have plenty. From an array perspective we are doing NOTHING, yet the DB tools shows high IO waits and we are seeing some of this on the system. The f/s we are exercising is spread over many fast disks, as stated the disk subsystem, EVA8k does not seem to be the problem.
I just need some other tool to give me I/O stats on a few selected disks we are testing, something that APPS "people" can understand.

[Hein van den Heuvel]

DB? Which DB? Oracle?
For Oracle, using timed_statistics and STATS_PACK is one of the best ways to get relevant IO timings.

The only possible gotcha it to realize that Oracle starts the clock before the IO, and (obviously?) had to stop it itself when the IO is done, do it has to be scheduled back to do that. If there is a lot of CPU contention then the oracle IO times may seem longer as they include the time to get scheduled back.

Personnaly I like to take some baselevel testing outside the DB just to set expectations. Often a simple 'dd -bs=8k -count=10000' to the raw device (NOT a buffered file) is good enough for that, but yo may prefer a more exhuastive test with something like 'iozone'.

And... use the numbers you have, no matter how simple. Like are you running at 100io/sec, 1000, or 10,000? How many spindles are there? How thick is the hose? What that, what do you expect the IO performance to be? If the actuals are in the ballpark, then you'll have to focus on avoiding/combining IOs more so that 'whining' about how slow they go.

Cheers,
Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Yogeeraj_1]

hi David,

what database and database version are you using?

there are several database tools that offer very comprehensive reports on performance..

revert
kind regards
yogeeraj

[DavidJ]

DB Sybase ASE12-5 and we are using the Sybase tools to measure what the DB's doing. I need to confirm/disprove from a Unix level what the Sybase performance monitoring tools are showing.
Systems RP8420 11i and RX8620 11.23 both connected to EVA8k 2G pipes autopath configured with SST, shortest service time, to the luns.

[Court Campbell]

Dave,

This is hard to measure. Between the dynamic buffer cache and the cache on the EVA it's hard to tell disk throughput.

You might look into using extract. You will need the midaemon to be running all the time. This will allow you to get data over time to use for trending, etc. Here is a useful link on using extract:

http://www.nws.noaa.gov/mdl/awips/aifmdocs/APPENDIX06.htm

It's towards the bottom of the page.

You may also look into using EVAPerf. It can show you what the EVA is doing.

If you use glance and see a lot of disk usage you might want to pin it down to which filesystems/disks are getting pounded. Maybe you can shift things around. I am not familiar with Sybase and it's setup, but hopefully I have given you a few ideas.

Good Luck!

[Sumit Ghosal]

Hi,

You can use "iostat" check man iostat.

thanks

[Steve Cooper]

Sorry for the plug, but we may have what you're looking for. Please check out DiskPerf for HP-UX at http://www.allegro.com/products/diskperf/hp9000/.

Thanks,

Steve

[Trevor Roddam_1]

David.

Just my 2cents.
I know where you're coming from.
I've done this with DBA's many times.

The best test is to prove the system can do the high IO. This can be as simple as a timed DD test. When they see the IO provided you can then ask "Does your app need more IO than this? Why does it need more? This is more IO than XYZ and refer back to the Sybase or their application docco."
This normally works. If not....

First, it would seem there are too many cache systems. Turn off as much of the LVM cache. Down to 10% max would be good and let the DB use it's cache.

The EVA wil do what it does and should be the last point of tweeking.

Try to use several LUNs from the EVA and either concatonate or stipe in LVM, this adds more Physical Drives" to LVM and it seems to like 4 - 16 depending on the TB's.

Using the GUI version of Glance, sar and the others mentioned above you will see the IO per LUN, using pvmove you can tweek the balance a little online to get more out of the system. If you woul like help on this let us know.

Using one huge LUN will cause IO bottle necks.

Hope it helps.
Trevor.

[DavidJ]

Thanks everyone for your assistance. I now have some ammo and my guns are loaded;-)

Trevor, some of your remarks brings back an old argument of should we strip on the EVA, should we allocate many small luns as the EVA spread the allocation accross x disks bla bla bla.......

We will not open this Pandora's box again ;-)

Once again, Thanks All.

[DavidJ]

Thread closed