iostat

[MohitAnchlia]

Below is sample output of iostat from our Prod env. What I am seeing is that when avgrq_sz goes up out latency also goes significantly up even out r/s and w/s are not that big. So in below output first 2 lines have desired latency but then next 2 lines have high latency even though r/s and w/s is the same.

My questions:

1. What drives avgrq-sz? Why is this high in one scenario? I am assuming that's driving the latency high.
2. How to improve avgrq-sz so that latency remains as expected?

I have some more question at the bottom.


Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

sdh 0.01 0.01 1.46 1.83 46.06 79.51 38.15 0.02 5.23 4.83 1.59
sdi 0.01 0.01 1.16 1.89 36.07 79.75 38.00 0.02 5.65 5.23 1.60
sdt 0.02 0.02 0.09 0.13 48.77 52.40 469.71 0.00 16.51 10.45 0.23
sdu 0.02 0.02 0.09 0.13 48.86 52.41 471.72 0.00 17.14 10.83 0.23



More question:

Definition of await is this "The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them." Does this mean it is wait time + service time. So if I did await - svctime I get the real wat time?

[Steven E. Protter]

Shalom,

1. Average read queue size is driven by a number of factors. If you have a lot of applications or clients needing to read data, this can cause this problem. Also a heavy write environment can slow down reads because the disk heads are busy.

2. Measure this based on application response time. If application response time is good, ignore these figures. If application response time is poor, look into factors that cans cause this.

Last question. Your formula is correct, but the await is calculated for you. It is a "real" number and there is no need to slice it up int components.

SEP

[MohitAnchlia]

We are using SAN and EMC storage. Is there a way to make things faster from storage point of view? I am trying to proactively look at this because we will have twice as much read and write as we enter our peak period. Any suggestions on reducing it from storage perspective will be helpful.

[MarkSeger]

If the application you're trying to speed up is well written and the system properly configured it may be doing as well as expected. Do you have any performance levels for the application to measure against? If so and you're meeting them, the first step is see if your storage is performing properly and that can be a lot of work.

The first step is to see what you baseline I/O rates are for various workloads and start from there. When doing this I favor using collectl, probably because I wrote it, but also because it shows me the types of things I'm most interested in and all on one line, rather than iostat which spreads things out on many lines and I personally find difficult to read, especially if I'm sampling at 1 second intervals which I always do with collectl. But the methodology is the same.

First start off verifying your storage is performing as expected by writing large files to it and watch the cpu/disk loads with collectl/iostat in another window and see what's happening. In the case of a single, local disk this is always pretty easy. But for a disk array or san this can be more difficult because you may need many clienta to generate the load, sometimes dozens or more! A tool like 'dt' lets you run multiple threads on a single system and that can help reduce the number of clients, but you still may need a lot of concurrent processes AND remember your client can never run faster than your connection to the storage!

Once you've satisfied yourself you're able to write at the levels expected, now do it for reads and these can take more clients than writes, something 2-4 times as many because of read latencies.

If your application is doing smaller I/Os, and most do, you need to try benchmarking these. If you're doing random I/O now you're paying the price of moving the disk heads and waiting an average of 1/2 of the rotational latency for each access which is MUCH slower than sequential I/O, typically on the order of only a few hundred reads/writes per spindle (and that's why you need raid!).

Only after you've made it this far can you then start to think about speeding things up for your application.

Hope this helps...
-mark

[MohitAnchlia]

How do I verify if we are mostly doing sequential or random IO? Application does so many things that it's difficult to discern. Is there a better way?

[MarkSeger]

Perhaps I got a little carried away with my explanation as it more general. I went back and looked at the numbers in your iostat, and from what I see the wait and service times are just fine. Having a bunch of requests in the queue is typically not that big an issue.

At the same time, you've only shown a single data point and that's only an instantaneous measurement. To get a better feel for whats going on you should take many measurements a one second intervals and watch the behavior.

I guess the real question behind all this do you think the application is running slower than it should be? If not, don't worry about if and if it is, what causes you to believe this.

-mark

-mark

[Steven E. Protter]

Shalom,

You ask:
How do I verify if we are mostly doing sequential or random IO? Application does so many things that it's difficult to discern. Is there a better way?

You need to look to the application.

Oracle archive logs are written sequentially. Most other oracle i/o is random.

Depends on the application.

You might wish to award people that help you with ITRC points. I don't need any but you might find it a motivating factor in getting better help.

SEP