Sar -d shows high device %busy but low avwait & avserv

[James Brand]

I once read in an HP performance tuning white paper that any disk over %50 busy was saturated. Recently I've noticed a few SAN disks with high %busy but decent average wait and service times such as this one:


device %busy avque r+w/s blks/s avwait avserv
c7t10d2 93.12 0.50 753 12052 0.00 1.28

sar at 60 second intervals, NOT using raw I/O.
So is this acceptable?

Thanks,
Jim

[A. Clay Stephenson]

This is one of those it "it depends" question. The fundamental problem is that sar (or Glance or any host-based performance tool) can't tell the difference between what is 1 physical disk (in which case, you probably should be concerned) and what is 1 logical disk which might be comprised of stripes across many physical disks (in which case, you probably shouldn't be concerned). Moreover, because your SAN LUN's are likely to be heavily cached, a great deal of the i/o might be taking place in the SAN's cache and thus be very fast.

[Steven E. Protter]

The SAN can be tuned. If its raid 5 parity 9 you are doing a lot of writes for every single write.

Before you start messing around, a more basic question should be asked. Is there a problem? Is there a complaint about performance of applications. If not, move on.

If yes, you might want to collect data over time.

This script: http://www.hpux.ws/system.perf.sh

Collects performance data over a longer period. Its easily configurable and covers areas other than disk. You'll get some good averages and realize that whats going on with this disk could just be a short term issue.

SEP

[Vincent Fleming · Accepted solution]

I've never found %busy to mean anything AT ALL in a SAN.

Your ave queue depth is 0.5, avwait of 0.00, and avserv of 1.28 - all sound wonderful.

It looks like you're doing 753 8K IOPS - which is not high for high-end disk arrays; most can handle several thousand IOPS at 8k easily.

So, don't worry about the %busy - it means nothing.

Be concerned when your avwait + avserv is more than 10 to 15 (depending on your environment) or when your queue depth is > 4 regularly. Usually these two events coincide.

Many places regularly run their systems with avwait + avserv approaching 30ms, but I like my systems more responsive than that, if possible.

Regards,

Vince

[James Brand]

Thanks guys, your responses helped a lot.

Jim

[James Brand]

Question answered.

Jim

[Zinky]

I have been using and monitoring SAN disks (cache or controller - centric ones) over the years and the ONLY statistic that you will need to watch whether your I/O subsystem is getting hammered or not is the queue statistic: sar -d (avqueue) and glance -u (Qlen column).

Any queue value observed above 1 is Bad and applications should already see a bottleneck.