topic 'avwait and avserv are normal but Disk is 100% busy on HP-UX lance' in Operating System - HP-UX

'avwait and avserv are normal but Disk is 100% busy on HP-UX lance'

abdul javeed — Wed, 09 Jun 2010 04:09:42 GMT

Hi Friends,

Disk is 100% busy. But avwait and avserv are normal. Please find the below sar and glance out put.

$ sar -d 1 10

HP-UX lance B.11.31 U ia64 06/08/10

11:44:26 device %busy avque r+w/s blks/s avwait avserv
11:44:27 disk25 100.00 0.50 987 19840 0.00 3.77
disk27 100.00 0.50 240 4032 0.00 5.34
disk47 24.00 0.50 43 688 0.00 5.66
11:44:28 disk3 1.00 0.50 5 48 0.00 1.83
disk25 100.00 0.50 1177 22912 0.00 3.27
disk27 100.00 0.50 285 5472 0.00 3.98
disk47 19.00 0.50 44 720 0.00 4.16
11:44:29 disk25 100.00 0.50 1198 21536 0.00 3.15
disk27 100.00 0.50 274 4896 0.00 3.82
disk47 13.00 0.50 28 448 0.00 4.89
11:44:30 disk25 100.00 0.50 1148 18376 0.00 3.35
disk27 84.85 0.50 191 3345 0.00 4.51
disk47 24.24 0.50 40 646 0.00 5.98
11:44:31 disk3 2.97 0.50 3 22 0.00 10.87
disk25 100.00 0.50 983 17743 0.00 4.04
disk27 87.13 0.50 168 2788 0.00 5.17
disk31 0.99 0.50 2 32 0.00 4.86
disk34 0.99 0.50 2 32 0.00 4.27
disk47 18.81 0.50 32 523 0.00 5.83
11:44:32 disk25 100.00 0.50 858 15760 0.00 4.89
disk27 84.00 0.50 142 2288 0.00 5.91
disk47 10.00 0.50 19 304 0.00 5.09
11:44:33 disk25 100.00 0.50 1098 21737 0.00 3.58
disk27 84.85 0.50 155 2796 0.00 5.64
disk47 4.04 0.50 8 129 0.00 4.96
11:44:34 disk3 0.98 0.50 1 8 0.01 6.95
disk25 100.00 0.50 1210 21380 0.00 3.26
disk27 86.27 0.50 177 2839 0.00 4.86
disk31 0.98 0.50 2 24 0.00 1.19
disk47 7.84 0.50 20 329 0.00 4.38
11:44:35 disk25 100.00 0.50 1503 26101 0.00 2.43
disk27 100.00 0.50 281 5010 0.00 3.89
disk47 3.03 0.50 5 81 0.00 5.94
11:44:36 disk3 0.99 0.50 2 20 0.00 3.65
disk25 100.00 0.50 1200 25236 0.00 3.04
disk27 93.07 0.50 236 4436 0.00 4.06
disk31 0.99 0.50 5 55 0.00 2.57
disk47 5.94 0.50 11 174 0.00 5.57

Average disk25 100.00 0.50 1136 21061 0.00 3.39
Average disk27 94.91 0.50 215 3788 0.00 4.57
Average disk47 12.99 0.50 25 404 0.00 5.21
Average disk3 0.60 0.50 1 10 0.00 5.09
Average disk31 0.30 0.50 1 11 0.00 2.77
Average disk34 0.10 0.50 0 3 0.00 4.27

----------------------------------------------

Glance o/p:

Glance C.04.70.001 11:43:46 lance ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU U | 24% 17% 24%
Disk Util F F |100% 100% 100%
Mem Util S SU U | 80% 80% 80%
Swap Util U UR R | 51% 51% 51%
------------------------------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 12
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 800% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
oracleorsblp 23121 oracle 60.3 1 0.9 39.6mb 46.5mb SOCKT
oracleorsblp 18003 oracle 37.1 1 239.4 248.2mb 256.6mb CACHE
oracleorsblp 20664 oracle 36.3 1 513.6 46.1mb 48.9mb CACHE
oracleorsblp 5522 oracle 14.1 1 9.6 34.0mb 38.3mb SOCKT
oracleorsblp 692 oracle 10.6 1 213.0 56.7mb 57.0mb CACHE
oracleorsblp 21039 oracle 7.6 1 207.1 34.4mb 41.6mb CACHE
oracleorsblp 22953 oracle 6.1 1 195.7 33.8mb 41.6mb VM
oracleorsblp 21666 oracle 3.6 1 191.9 33.3mb 41.4mb CACHE
vhand 2 root 3.4 1 0.5 64kb 72kb SLEEP
glance 23153 oracle 2.9 1 0.0 13.0mb 18.5mb STRMS
emdctl 23157 oracle 1.7 1 0.1 92kb 212kb died
midaemon 2397 root 0.4 6 0.0 135.3mb 139.4mb SLEEP
ora_mmnl_ors 25408 oracle 0.2 1 0.0 32.5mb 282.8mb OTHER
emagent 22234 oracle 0.2 8 0.0 100.7mb 147.5mb SLEEP
oracleorsblp 12250 oracle 0.0 1 0.0 42.7mb 46.8mb SOCKT
oracleorsblp 11298 oracle 0.0 1 0.0 43.0mb 57.2mb SOCKT
oracleorsblp 21002 oracle 0.0 1 0.0 33.6mb 38.5mb SOCKT
oracleorsblp 3520 oracle 0.0 1 0.0 24.0mb 28.7mb SOCKT
oracleorsblp 11433 oracle 0.0 1 0.0 58.2mb 62.7mb SOCKT
ora_pmon_orc 24466 oracle 0.0 1 0.0 260.6mb 362.6mb SLEEP
ora_pmon_ors 25382 oracle 0.0 1 0.0 33.5mb 314.7mb SLEEP
oracleorsblp 10785 oracle 0.0 1 0.0 81.2mb 85.7mb SOCKT
oracleorsblp 21622 oracle 0.0 1 0.0 33.9mb 37.0mb SOCKT
ora_p000_ors 7903 oracle 0.0 1 0.0 33.3mb 69.1mb OTHER
oracleorsblp 10342 oracle 0.0 1 0.0 89.9mb 94.3mb SOCKT
oracleorsblp 10582 oracle 0.0 1 0.0 70.2mb 74.5mb SOCKT
oracleorsblp 20643 oracle 0.0 1 0.0 34.8mb 42.9mb SOCKT
oracleorsblp 22682 oracle 0.0 1 0.0 34.1mb 43.4mb SOCKT
oracleorsblp 10787 oracle 0.0 1 0.0 34.6mb 34.7mb SOCKT
ora_dbw3_ors 25394 oracle 0.0 1 0.3 37.9mb 38.1mb OTHER
oracleorsblp 22079 oracle 0.0 1 0.0 33.3mb 37.4mb SOCKT
oracleorsblp 11252 oracle 0.0 1 0.0 59.8mb 69.9mb SOCKT
oracleorsblp 11865 oracle 0.0 1 0.0 31.8mb 49.0mb SOCKT

Re: 'avwait and avserv are normal but Disk is 100% busy on HP-UX lance'

Kapil Jha — Wed, 09 Jun 2010 06:00:38 GMT

check in glance which disk is using high IO.
You can see the stats for file system and disk.

Then see this disk or file system is used for what.

act accordingly.

BR,
Kapil+

Re: 'avwait and avserv are normal but Disk is 100% busy on HP-UX lance'

abdul javeed — Wed, 09 Jun 2010 06:22:53 GMT

Siebel application is running on it.

As per my understanding the high IO disks are disk 25 and disk 27

Re: 'avwait and avserv are normal but Disk is 100% busy on HP-UX lance'

chris huys_4 — Thu, 10 Jun 2010 09:01:13 GMT

Hi,

Double up the scsi_queue_depth parameter for each of the 100% disks, until the disks shows values lower then 100%.

For more explanation of the reasons behind the doubling up.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1433815

Only unlike the command that was used, in the above thread, to double up the scsi_queue_depth parameter, i.e. the scsictl command, on HP-UX 11.31, the scsimgr command should be used, to increase the max_q_depth (the 11.31 "name" for scsi_queue_depth).

See also the scsimgr whitepaper, on how to increase variables.

http://docs.hp.com/en/scsimgr/scsimgr_whp_AR0709.pdf

i.e.
1. Check current value max_q_depth for disk25/disk27

# scsimgr get_attr -D /dev/rdisk/disk25 -a state -a max_q_depth

# scsimgr get_attr -D /dev/rdisk/disk27 -a state -a max_q_depth

2. Double up max_q_depth for disk25/disk27

# scsimgr save_attr -D /dev/rdisk/disk25 -a max_q_depth=(value_found_in_"1"_for_disk25 *2)

# scsimgr save_attr -D /dev/rdisk/disk27 -a max_q_depth=(value_found_in_"1"_for_disk25 *2)

3. check if max_q_depth was increased.

# scsimgr get_attr -D /dev/rdisk/disk25 -a state -a max_q_depth

# scsimgr get_attr -D /dev/rdisk/disk27 -a state -a max_q_depth

4. check with sar -LdR 1 10, "-LdR" to see not only total IO per disk per second, but also total read and write IO per disk per second, if the 100% is gone.. If not double up the value of max_q_depth again, and this until the 100% is no longer there (and check also if the amount of total IOs performed by the disks in 1 second gets higher and higher)..

NOTE: A lot of very fast "physical read IO's" supplied from the diskarray, could mean that caching on database/application level, is not adequately configured as those read IOs probably then come from the diskarray cache, vs read from "diskarray disk", and were does allready requested once before by the database/application...

Greetz,
Chris