Disk performance issue

[rana786]

Hi All,

I need the clarification of following issues. PLease look at the below output.

Nine hour disk IO status of blbill02 on 1st june 07

device %busy avque r+w/s blks/s avwait avserv
Average c9t0d0 47.23 18.25 256 7247 3.58 2.22
Average c9t0d2 1.56 0.5 9 273 0 2.45
Average c12t1d0 0.46 0.5 4 161 0 1.39
Average c9t1d1 1.71 0.5 15 691 0.02 1.52
Average c9t1d7 2.26 0.5 25 1117 0 1.05
Average c12t2d7 2.29 0.5 44 1745 0.01 0.73
Average c12t0d1 53.87 2.55 322 11866 0.33 2.15
Average c12t1d6 7.11 51.04 64 2495 8.03 1.87

For disk c9t0d0, it is 47.23 %busy, avwait 3.58 and avserv 2.22. On the other hand for disk c12t1d6, it is only 7.11 %busy but avwait is 8.03 avserve is 1.87. What is the reason behind it. Please help by clarifying it.

Best regards,
Rana

[Ninad_1 · Accepted solution]

Generally I have seen this type of results from burst of small reads physically spread across disks.
Whereas if you have sequetial reads, the output is more like the c9t0d0 output you have shown.

Regards,
Ninad

[rana786]

Hi Ninad,

Thanks for your prompt response. Could you please explain a little bit about avque value of said two disks?

Best regards,
Rana

[Ludovic Derlyn]

hi,

avque indicates the average percent number of requests in the disks I/O queue

Disk controler appears to have one busy drives c9t0d0 , same item for c12t0d1

What is your fs localized onthe two disks ?

have you JFs online installed on your servers ?

Regards

L-DERLYN

[Ninad_1]

The avqueue is indicating that though the disk c12t1d6 is less busy compared to c9t0d0, the requests are queuing on this disk more which is not a bery good sign if the avqueue starts going higher.
This could be due to say lot of small files on the disk along with application behavious for the requests. Its really hard to pin point to a very accurate judgement not knowing who is using the disk, what are the applications etc.

Regards,
Ninad

[rana786]

Hi All,

The application running on these disk is Oracle TABs and the file system is raw. Its a RAC cluster with raw file system using EMC storage.

Best regards,
Rana

[Hein van den Heuvel]

Specifically with RAW device ORacle can easily issue multiple IOs to the same device in burst.
Outside the burst it is not used much --> low busy%.
During the bursts the driver may choose not to pass all IOs on the the controller causing high avwait. If it did pass them on, you'd likely see the waiting moving from the server to the storage and the end result might just be lower avwait but higher avserv, or could indeed be better.

You may want to google for : scsi_max_qdepth

If that's set too low (the default = 8 = too low imho), then IOs will be held up in the server instead of utilizing the power of the Storage subsystem.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[rana786]

Hi Heuvel,

Thanks for your details. BTW, can you tell me what should be the optimum value of scsi_max_qdepth for utilizing the power of storage subsystem.

Best regards,
Rana

[Tim Nelson]

There are a couple papers from HPs end about scsi_queue_depth. Search this forum for it.

All in all it you can increase the OS to be more robust. I have not seen any benefit though in my testing. The old rule was a queue depth of 8 for a single disk. With an array there typically is multiple disks in play and therefore could be 8 per disk.. I.e. an 8 disk stripe could handle 8*8 requests. Hence a change the depth to 64. The HP doc refereces XP configurations specifically..

If there are a number of disk supporting each device then set the depth to 16 ( i.e. double ) and gather stats.

Again. I have not seen any real reduction in queues with my testing, but....

With the service time being so low and the number of requests also low, I expect that this is more an issue with low volume small block reads and the application continuously going after the same blocks.

[rana786]

Thanks all.