io queues

[Tim Nelson]

I am looking into some io events and hoping someone would like to comment.
I am seeing through Polycenter Capacity planner a some large io queue lenths at times. While investigating there seem to be a number of items that are odd.
1) during the time of large queue lenths for a disk the io/s is not extremely high. There are other times during the day where the io/s is much higher and the queue lenth is low or zero. I also do not see any other disks with
2) viewing the statistics on the disk end ( EMC Symmetrix) the disk utility and response time is very low.
Any input on why a queue lenth would be high at times when it seems that the channel and disk are any busier before, during or after?
Thanks in advance for any inputs.

[John Gillings · Accepted solution]

Tim,
Many possible reasons. The I/Os might be large, have poor locality or are bypassing cache. Perhaps there's a BACKUP in progress?

The only way to say for sure is to observe the system at the time and see what processes are involved and what they're doing.

That said... make sure you've got an actual problem before you spend a lot of time and resources trying to solve it.

[Tim Nelson]

Thanks for the note John. I hear ya..
Just happened to see the high queues ( i.e. 400+) and the red flag went up.

[Ian Miller.]

as John said BACKUP would be favorite to do this as it queues as many I/O requests as possible to a disk. You may be able to get a top 10 images doing direct I/O sort of display.

[Richard W Hunt]

When you said "Symmetrix" my ears perked up. There are times that we have problems because our Symm often proves that it has a mind of its own.

If you are doing continuous replication to your backup system (presuming you have one), check also for networking delays from the primary Symm to the backup system. In continuous mode, some operations have to be synchronous, and if you have a momentary network hiccup, they have to be delayed.

If you are doing point-in-time form of replication, this is less likely to be an issue because in that mode, the data path is not synchronous.

[Tim Nelson]

I am thinking this issue is directly related to the application and its attempt to update a single file system resource ( i.e. file structure, index, lock, etc ) and sending a couple hundred/thousand of these updates all at once. Hence low disk io but long queue. I have seen this before in a Universe environment where the application was told to update a specific file structure over 10,000+ times in a loop, the queue at the disk level was over 30,000 and I gave the progammer a six pack for being able to drive the queue so high.