High values in sar's %wio

[Stojcevski Dejan]

I have a rp2450 with 1GB of ram and 2 CPU's on 550MHz. Attached is a SC10 disk system with 18GB disks on 15k rpm. sar -d output shows ok results (0.5 on queues and around 10ms in average service time).
I have a high values in %wio output of sar (attachment). There is a Informix db running on the machine.
By tuning Informix no sighificant improvement has been shown in the sar.
Please help. I do not know from where I should start!

[Steven E. Protter]

You need to look at whats happening. It seems you have bursts of activity that is causing the higher wait times.

This usually indicates a bottleneck. Whether to proceed further though depends on this: Does the problem prevent you from using the system? High wait times by themselves are not a reason to do anything, especially when its transient.

You may wish to revisit how you are laying out your data and index files across the disk.

SEP

[Stojcevski Dejan]

Data is equally spread across all the disks. We are using raw chunks for the database (no cooked space at all for the chunks!). There are around 15 lvolumes created with -D y option which means all of the space is equally spread across all the disks. All of the lvol's are mirrored as well across two physicall groups. Mirror copies = 1.
That said I beleave the config is ok. What else should I pay attention on?
Dejan.

[Jeff Schussele]

Hi,

I'd check the cron entries & talk to the DBAs to see if they schedule anything at those times because your values go high at midnight and noon only.
Sure looks like some kind of reporting or heavy querying going on - especially if it's the same *every* day.

My 2 cents,
Jeff

[Stojcevski Dejan]

Forget those periods - it's backup and performance optimization of the database (update statistics in Informix)!
What I am talking is regular high %wio during the day. What should I pay attention to? What bittleneck it refers to? SCSI queue is not properly tuned? Why CPU can not cope with the bursts of workload?
Dejan

[Jeff Schussele · Accepted solution]

OK - let's focus on what the rest of the CPU usage is. Which is higher - User or System?
If it's *consistently* User then the problem is raw load & that can be many things like crappy SQL-type code or even simply not enough processor horsepower or many other things.
If it's system then you need to break it down even further using MeasureWare (now known as OVPA - OpenView Performance Agent). First look for serious context switching (GBL_CPU_CSWITCH_UTIL) this could indicate a VERY high fork rate and can be spotted with PROC_FORK. If you not forking a lot then this could indicate semaphore contention which mighte be handled by lengthening timeslice.
Back to System time - if it's high then start looking at system call rates (GBL_CPU_SYSCALL_UTIL) if higher than 30% then you need to see just *which* calls are chewing up CPU time. Use gpm & go to Reports->System Info->System Calls and sort on syscall rate or CPU time because some calls (Read, etc.) are much more expensive than others.
Finally it could boil down to a spinlock situation whereby all other CPUs are waiting on a resource that can only be accessed by a single CPU at a time. This type thing is best traced down with a utility like tusc.
But never lose sight of the fact that it could simply be that an SC10 and the HBAs handling it are just going to carry the load & it's time to step up to a fibre channel SAN with multiple FC cards to spread the load. Only after you rule out CPU & code problems can you be certain you need faster disk I/O.

HTH,
Jeff

[Stojcevski Dejan]

User part is ALWAYS greater that system one. GBL_CPY_SYSCALL_UTIL is also low - around 2%. I guess that means physical limitations of the system - time to look forward to FC ...
Still, if I can manage to increase the buffer pool for DB so data to stay in memory instead on disks I guess I will lower IO's and achieve lower %wio. But that is different storry.
Thanks again for your time.

[Jeff Schussele]

Yes - beginning to sound like a buffering issue & I'd be reluctant to increase dbc_max_pct as this could cause "double-buffering" and actually slow up I/O.
And since I'm an Oracle guy & not Informix I'll defer to someone who might be able to help you increase the Informix cache size.

Good Luck,
Jeff

[Stojcevski Dejan]

Thanks.