What tools can help identify RAID as source of bottleneck?

[Debbie Fleith]

We have a new Linux server with a RAID5 / 4 disk configuration and we suspect that may be causing some performance issues. But, how can I prove that? Our hardware vender informed me they don't have any tools to analyze the RAID performance. Can any sar options point to the problem? If so, how can I analyze sar output to pinpoint RAID issues?

[Ivan Ferreira]

vmstat willl provide you the cpu "wa" (wait) information. Normally, cpu wait are caused by pending I/O operations to disk, meaning that the disk subsistem is slow (or a poorly designed application).

Then, you could do performance test to identify the maximum transfer rate to the disk subsistem, for example using dd. Then, identify if your application can reach that transfer rate.

[Debbie Fleith]

Is there an easy way to run vmstat every 5 minutes or do I just setup a cron job to run it on the frequency I want?

[Ivan Ferreira]

The vmstat command receives as argument the interval and the number of samples:

vmstat [interval] [number_of_samples]

For example, using:

vmstat 300 10

Will display 10 samples every 5 minutes (300 seconds).

[Debbie Fleith]

What is considered a high wait figure with vmstat?

[Ivan Ferreira]

On my systems they stay close to 0. A small ammount < 5 is acceptable for small periods of time.

But this is system dependent. As a general rule, a value close to 10 can indicate an I/O problem. As a rule of thumb, consider 25% as high iowait if you have idle cpu.

Iowait can be caused also by paging/swapping acitivities, this occurs if your system memory is not enough, check the swap in swap out columns in the output of vmstat (si/so).

You can also use the "sar -d" and iostat commands to identify the transfers/request per second.

You should also see:

http://www.sarcheck.com/
http://www.linuxquestions.org/linux/answers/Jeremys_Magazine_Articles/Hunting_I_O_Bottlenecks_with_iostat