Strange result of disk striping

[Ian Dennison_1]

I have a strange result of striping an Oracle database on an XP512. I am using a 1MB Stripe with 8 Spindles, on a fairly active database, yet when I measure the results through Glance, Process IO wait times (actual and %) have increased not decreased.

I am measuring the following stats from measureware,..

PROC_DISK_SUBSYSTEM_WAIT_TIME
PROC_DISK_SUBSYSTEM_WAIT_PCT
PROC_OTHER_IO_WAIT_TIME
PROC_OTHER_IO_WAIT_PCT

I total up each parameter by day, and count those that have values > 0. I then average this out for the following,....

% of Processes Non-Zero Records
Average Wait Time for Non-Zero Records
Average Wait % for Non-Zero Records

Day Metric Total Total Total NonZero Ratio Average Average
Num Name Time Percen Entries Records Records Time Percent
--------------------------------------------------------------------------------
04 oracleSID 2476143 15589.96 3228 2468 76 1003.30 6.32
05 oracleSID 2234530 9647.79 1999 1408 70 1587.02 6.85
06 oracleSID 5 28.51 1 1 100 5.95 28.51
07 oracleSID 12237 145.15 7 7 100 1748.27 20.74
08 oracleSID 2516713 17244.12 1470 1464 99 1719.07 11.78
09 oracleSID 5921873 21323.46 3216 1237 38 4787.29 17.24

I am absoultely stumped as to why this is occuring. Has anyone else any ideas?

Stepping away from the immediate problem, how can I best show the hierarchy that my striping exercise has shown gains at the OS level using OS Statistics?

Thanks. Ian Dennison

PS I have been on the HP-UX Performance and Tuning Course, and reinforced this with experience.

[Stefan Farrelly]

The best way to show striping speed improvements is with dd.

eg;
time dd if=/dev/vgXX/rlvolYY of=dev/null bs=1024k count=100

This will read 100Mb and tell you how many secs it took. Then try it on another lvol thats NOT striped and compare. You will see a big difference.

[Tom Geudens]

Hi Ian,
I'm a great fan of using IoZone to do the tests (before and after).
www.iozone.org

Regards,
Tom

[Ian Dennison_1]

Stefan / Tom,

Thanks for the info; I have bookmarked the appropriate sites.

Unfortunately I have to prove that benefits have been shown to this particular database. Management are confident that striping works in theory; while the tools you have mentioned reinforce that theory, they do not show the dierct gains made for the User in this specific instance (no pun intended).

The more I think about it, the more I lean towards measuring the performance within the Application.

Ian Dennison

[Pete Randall]

Not to rain on your parade but it is possible that the database was well distributed to start with and striping has ended up unbalancing that nice even distribution.

Pete

[Tom Danzig]

A 1MB stripe is a bit large IMO. I usually make sure the database block size and file system block size are the same (8192) and use a 32KB stripe. Others have used a 64K stripe size as well.

The tests I ran using iozone and dd showed striping to be faster than non-striped. The difference between the 32KB and 64KB stripe was insignificant. You may want to adjust the stripe size down if you can and test again.

[Stefan Farrelly · Accepted solution]

You asked for ways to measure the gains at OS level originally. Now, if you want to test it application wise then you need to really hammer the disks to show the improvement.

The best example I can remember is a huge oracle job which used to take several hours run over an evening, after moving to a striped lvol it shaved over 4 hours off it. It was an i/o intensive report. The DBA's were impressed.

Its no good measuring such simple metrics as wait times, as these wont fluctuate much (and as you have shown they went up!). They probably went up because now with several spindles needto be accessed for an i/o request whereas before only one spindle. To really get the benefit of striping, and show it, hammer your disks to the max. Run an enormouse query or report which takes hours. This will prove conclusively that striping at your application level is faster.

[Ian Dennison_1]

Looks like it is back to the drawing board; this time, with tests and benchmarks before and afterwards.

Thanks all for your help. ian

[Tim D Fulford]

Ian

Lots of good ideas above. Personally I would step back & look at the target you want to hit!!! on the what can I look at front, I use MeasureWare to do the following

o For each disk in the stripe set plot utilisation Vs time - Overlay these results. If there is a large difference between the busiest disk & the quietest, your stripe size is too fat.
o For each disk plot IO/s Vs disk%. A majority of the points should lie on a line. this will approximate to the service times for the disk. The actual calculation is
avg ser [ms] ~ disk% * 10 / IO Rate
o Similarly plot disk kB Vs disk %. Again you should get a fairly straight line.
o From the above two you can get an idea of the kB/IO, and an estimate of peak IO & peak kB throughput.
o Plot kB/IO Vs time. Look at the most common block size, this should be the stripe width (well you may need to round it up to the next available size)

Repeating of the above advice, 1MB stripe size seems very big. You will need to be doing IO's of 8MB+ to get the full bebefit from this stripe width & length. Oracle uses, I believe, 4 or 8kB page size. I would reduce the stripe size IF the disks still have hot spots. You can also think about increasing the stripe length over more disks if you want to keep the full stripe the same.

The other thing about stripe size is that a large block/stripe width is more efficient than a very small one. Whilst this is true, if you over do it and only need a few kB from each IO, then you are choking your system with essentially unwanted information. What you could do is determin the most common IO size & maybe round UP, so if you see 2.5kB, round to 4kB.

At the risk of teaching you to suck eggs, if you double the number of spindles you should be able to double the Max IO & kB throughput, or reduce the activity of the disk by 2. If this does not happen then something has gone wrong (as I have seen with a customer who used a 64kB stripe on a system that really neded 4-8kB).

Regards

Tim

[Ian Dennison_1]

Tim,

Apologies for not applying points sooner. Magic answer, information gained from the rules you gave has proved invaluable.

Thanks. Ian