Disk bottlenecks/ performance.

[John Tyler]

Hello!

We have a N-Class server connected to our SAN running a database. There has been complaints that system is slow.

I looked at the sar numbers. The system is busy with $usr averaging 60. But %wait i/o is zero.

sar -d shows some of the disks are 100% busy.

Since %wait i/o is zero, Can i assume that,though the disks are busy, the system is able to complete the i/o on time.

I would appreciate any suggestion.
thanks.
John.

[Shannon Petry]

Looking at the output from sar, it looks like this system is not configured for optimal disk I/O. It looks like everything is configured on a sigle disk that is currently in use.

How was the system created? lvols, disks, etc... and what is it used for?

Regards,
Shannon

[John Payne_2]

John,

"The system is slow" is pretty vague. Can they give you any more information than that?

Has the system always been 'slow'? If not, what changed?

Is there a possiblity that you have a HW failure somewhere down the path to your disks? (I have seen this on our EVA. One of the paths were failed and because load balancing was not happening, our DBA's said "Things are slow." I do so love that phrase...)

If it's always been 'slow' and there are no HW issues, we will need more information, please.

Hope it helps

John

[John Bolene · Accepted solution]

100% busy is not a problem with a VA or a SAN as measureware does not collect valid data for fast devices.

What concerns me is the avque, avwait, and avserv of drive c9t0d1. This is much too high and is probably causing the perceived slowdowns.

Avque should be 0 or 1, avwait should be close to 0, avserv should be way lower than 30.

[Bill McNAMARA_1]

run
iostat -t 5
to check the disk usage.
make sure you balance loads appropriatly with lvm and lvol configuration.
also check your sar patch level.

[Michael Steele_2]

That c9t0d1 avque and avwait are way, way out. If this is consistent throughout the day then you've got a huge I/O bottleneck issue. 'c9t0d3' also tests for a disk bottleneck at one point but not an I/O.

Note the 'c9' controller will be one HBA and have a device like /dev/td0, etc.

Regarding $wio, this has to be over 15.

Need more information about your type of disk array (* xp512? *), SAN topology, (* arbitrated loop ? *), model of HBA (* A5158A? *), any SAN switches?

Please attach these especially your 'fcmsutil' report.

ioscan -fnkC fc
fcmsutil /dev/td# state
lvdisplay -v
vgdisplay -v

[A. Clay Stephenson]

You clearly have a disk bottleneck which suggests that you need to divide and conquer. Sar is not a really good tool for this; Glance is better but even there, there are limits. All your host knows is that a tremendous amount of I/O is going through what it sees as one device. It might be many physical devices - sar (or Glance) just doesn't know. Your disk array should come with additional tools in helping you identify the hotspots. You should be able to create additional LUN's with different I/O paths.

Now here's the bad news. Divide and conquer may only help a bit. You numbers are so bad that even dividing by four - you're still lousy. This strongly suggests that what is really needed is some careful SQL tuning.

[Stuart Abramson_2]

1. Your avserv is way too high. A good number for EMC disks that rotate at 10,000 RPM is 6 ms. Higher is bad. You have to check what kind of disks you have and how fast they rotate.

2. All your I/O seems to go on c9, but I bet that your disks are dual access on c9 and c10. You should alternate the primary and alternate more. (50/50).

3. avqueue higher than 3 is bad, and you have high avque on c9t0d1. You have too much i/o on c9t0d1, which probably means that you have "hot" files there, like oracle redo logs or archive logs. Move them around.

100% busy is not bad if avque and avserv are good. It means the disks are working hard, but within their capacity. When avque and avserv get high, then you have a bottleneck.

[Steven E. Protter]

In my far from humble opinion, there are a few things you should do:

1) Plan your disk layout a little better. You can't have too many oracle databases on the one virtual/physical disk. Keep index and data away from each other.

2) You can improve disk performance on the SAN by going with Raid 10. This will provide a significant bump in performance capacity.

SEP

[John Tyler]

Thanks for the many responses!

The diskframe is a HDS7700E.
When the server was setup, HP/UX was restricted to 8 luns. So these large size luns were setup. Now that restriction is fixed.

So i plan to get more LUNs added and also spread i/o over both c9 and c10.

Thanks
John.