topic Re: Disk queue high with low disk utilization in Operating System - HP-UX

Disk queue high with low disk utilization

Jack Fan — Tue, 19 Aug 2003 00:09:21 GMT

All,

Can someone help me to explain this problem and how can I solve it?

My HW specification,
Server : rp7410/8 cpu/16g ram
Disk Array : EMC 3330-18 / 18GB*20
DB : ORACLE 7.3.4.5
ERP : BaanIII

Before this ERP system migrate from HP V2500 server (12 cpu/12G RAM) to rp7410 (8cpu/16GB RAM), everthing is okay and performance good..
But after migration, the performance is very bad ....

Why we get high disk queue ( > 5) but the disk utilization around 80~90?

Any suggestion?

Re: Disk queue high with low disk utilization

Michael Steele_2 — Tue, 19 Aug 2003 00:21:04 GMT

Please post the results of the following:

vmstat 5 5
sar -d 5 5
sar -v 5 5
sar -u 5 5
swapinfo -tam
fcmsutil /dev/td? (* I/O reads and writes *)
queue_depth (* use power link *)
ioscan -fnkC disk

Re: Disk queue high with low disk utilization

Tim Adamson_1 — Tue, 19 Aug 2003 00:25:56 GMT

Also post kmtune, sysdef and cat /stand/system

Tim

Re: Disk queue high with low disk utilization

Jack Fan — Tue, 19 Aug 2003 00:58:01 GMT

Sir,

Kmtune & sysdef & system here, please check the attached file (zip).

Thanks,

Jack Fan

Re: Disk queue high with low disk utilization

Jack Fan — Tue, 19 Aug 2003 01:09:38 GMT

Sir,

sar / fcmsutil /ioscan results here, please check attached file.

Thanks,

Jack Fan

Re: Disk queue high with low disk utilization

Michael Steele_2 — Tue, 19 Aug 2003 11:08:30 GMT

If you refer to sar -d and check %busy you'll see EMS luns with values greater than 50% indicating a disk bottleneck. Your %wio greater then 15 values support this.

I/O under 'fcmsutil' is not listed and I am susptect all you I/O is going out one HBA. I.e., td0, td1, td2, etc. To correct this use a round robin PV Link ordering: c9 primary c10 alternat , c10 primary c9 alternate, etc.

You can also use LVM stripping to load balance but try round robin first.

Re: Disk queue high with low disk utilization

Michael Kelly_5 — Tue, 19 Aug 2003 15:24:32 GMT

The average service (avserv) times for your disks appear to be quite high. This would explain why your disk utilisation is is high.
You can see that there appears to be a correlation between the %busy and avserv figures (the more time it takes the disk to complete each request, the more time it is busy).
I would start by checking the disk array configuration.
Greater than 20 milliseconds to complete a disk request is an indication of a problem somewhere in the disk subsystem.
Perhaps caching has been turned off?

HTH,
Michael.

Re: Disk queue high with low disk utilization

Alzhy — Tue, 19 Aug 2003 15:34:09 GMT

Stripe accross your LUNS on different HBA's. Whether you are using LVM or VxVM you should be able to stripe accross... but put your redo logs on a non-striped LUN. Other things, check your mount options so you minimise buffering -- VXFS mount option convosync, direct??...

You may also check your throttling configuration and maxIO kernel parameters. What about your block sizes == is it equal to your Oracle DB Block size? VxFS block size should be 8192 (8k) which should match Oracle's DB Block size of 8192... as well..

Re: Disk queue high with low disk utilization

Tim Sanko — Tue, 19 Aug 2003 15:34:29 GMT

Two words "striped metas" from EMC if you are also short on EMC cache.

Do you have workload analyzer on your server?

If you have EMCs PowerPath you could eliminate the bottlenecks by adding an fibre HBA if you have an available slot in the 3830.

If you had a failure and are using PowerPath you would see high IO on one path and the IO would deadlock. Since Powerpath handles failover you might miss it...

Do a powermt display dev=all | grep dead.

If you have multiple HBAs and have lost paths this will explain high IO rates and this may be the problem we saw the same symptoms...

I have a V2600 production box and have 7410 for a test box. If you use pvlinks and a path failed you could see this....