sar -d can someone explain me this...

[Jonathan Caplette_1]

Hello,

I'm not that familiar with performance management on a HP-UX Box...

This box is running HP-UX 11.11, some users are complaining that there Peoplesoft jobs are very slower that normally. Peoplsoft is querying Oracle, those filesystems that are containing those apps, are on SAN disk (Hitachi). The applications managers are seeing disk bottleneck in Glance and its constantly at 100%...

I ran a sar for a couple of sec here is the output:

# sar -d 5 12

HP-UX psa2h01 B.11.11 U 9000/800 04/03/08

14:28:18 device %busy avque r+w/s blks/s avwait avserv
14:28:23 c0t6d0 4.79 0.50 13 101 0.00 4.55
c2t6d0 0.80 0.50 9 65 0.00 1.04
c22t1d1 74.45 0.58 156 5914 0.22 10.36
c22t1d3 72.65 0.59 190 6357 0.19 8.61
c22t1d5 91.62 1.00 996 72578 0.63 4.63
c22t2d1 0.60 0.50 1 18 0.00 15.00
c22t0d2 2.20 0.50 3 33 0.00 6.31
c20t1d2 96.21 0.72 311 8818 0.28 7.31
c20t1d4 81.04 1.12 578 11497 1.91 6.06
c20t1d6 63.47 0.76 571 30598 0.25 4.18
c20t2d0 34.33 0.50 64 818 0.00 6.60
c20t4d0 2.79 0.50 13 233 0.00 2.24
14:28:28 c0t6d0 8.22 0.50 42 362 0.00 2.44
c2t6d0 1.40 0.50 12 99 0.00 1.52
c22t1d1 56.91 0.54 105 3650 0.15 9.90
c22t1d3 99.80 0.73 312 9260 0.53 9.47
c22t1d5 79.36 0.94 760 36957 0.39 4.27
c22t0d2 0.40 0.50 1 8 0.00 6.61
c20t1d2 66.33 0.57 141 4162 0.18 7.91
c20t1d4 91.38 1.84 856 25472 1.71 5.53
c20t1d6 41.88 0.88 249 13983 1.19 5.05
c20t2d0 11.22 0.50 46 757 0.00 3.42
c20t4d0 1.20 0.50 9 164 0.00 1.46
14:28:33 c0t6d0 5.39 0.50 26 137 0.00 2.29
c2t6d0 1.20 0.50 22 105 0.00 0.62
c22t6d7 0.40 0.50 0 0 0.00 19.76
c22t1d1 89.42 0.55 150 3304 0.51 13.89
c22t1d3 99.00 1.02 408 16144 0.98 11.18
c22t1d5 82.24 1.15 489 18702 1.21 6.80
c22t2d1 0.20 0.50 0 3 0.00 3.23
c20t4d1 0.20 0.50 0 0 0.00 13.60
c20t6d7 0.40 0.50 0 0 0.00 20.27
c20t7d0 0.20 0.50 0 0 0.00 19.00
c20t1d2 58.68 0.61 97 3411 0.36 10.55
c20t1d4 83.23 1.68 717 21576 1.63 5.78
c20t1d6 39.12 0.51 262 15387 0.02 3.80
c20t2d0 5.59 0.50 55 1384 0.00 1.59
c20t4d0 0.60 0.50 4 102 0.00 1.45
14:28:38 c0t6d0 4.21 0.50 13 89 0.00 4.54
c2t6d0 1.00 0.50 9 70 0.00 1.12
c22t1d1 92.18 0.61 190 4366 0.33 10.63
c22t1d3 100.00 0.52 150 2674 0.05 11.21
c22t1d5 68.94 0.98 299 17129 1.17 8.53
c22t0d2 1.40 0.50 1 12 0.00 10.68
c20t1d2 65.93 0.52 452 18809 0.15 3.78
c20t1d4 90.78 2.24 598 14708 2.65 7.54
c20t1d6 38.08 0.50 223 12893 0.02 4.45
c20t2d0 6.81 0.50 14 176 0.00 5.34
c20t4d0 4.41 0.50 14 257 0.00 3.12
14:28:43 c0t6d0 3.79 0.50 9 73 0.00 5.77
c2t6d0 1.00 0.50 7 62 0.00 2.06
c22t4d2 0.20 0.50 0 0 0.00 9.80
c22t1d1 76.45 0.53 154 3248 0.31 9.90
c22t1d3 99.40 0.76 244 8105 0.78 13.06
c22t1d5 82.63 1.36 444 15946 1.32 7.42
c22t0d2 0.40 0.50 1 7 0.00 5.48
c20t4d2 0.40 0.50 0 0 0.00 12.79
c20t1d2 93.41 0.67 1097 50213 0.20 3.38
c20t1d4 93.81 2.51 781 19455 2.53 6.46
c20t1d6 78.64 0.54 350 18057 0.10 7.07
c20t2d0 23.15 0.50 38 505 0.00 6.47
c20t4d0 1.40 0.50 6 131 0.00 2.67
14:28:48 c0t6d0 3.40 0.50 13 92 0.00 3.21
c2t6d0 0.60 0.50 9 55 0.00 0.65
c22t1d1 86.60 0.54 159 3447 0.34 12.07
c22t1d3 100.00 1.22 429 16458 1.18 11.38
c22t1d5 91.60 2.10 430 17859 2.62 10.37
c22t0d2 0.60 0.50 0 3 0.00 34.73
c20t1d2 89.60 0.54 728 28941 0.09 3.33
c20t1d4 82.20 2.50 620 15878 2.71 6.82
c20t1d6 62.40 0.55 288 16381 0.08 6.02
c20t2d0 0.80 0.50 3 31 0.00 3.90
c20t4d0 0.80 0.50 4 90 0.00 2.17
14:28:53 c0t6d0 3.19 0.50 10 62 0.00 5.02
c2t6d0 1.20 0.50 7 52 0.00 2.12
c22t1d1 87.82 0.56 222 6899 0.41 10.15
c22t1d3 97.01 0.84 464 17910 0.57 8.34
c22t1d5 91.22 1.54 434 19203 2.55 9.22
c22t0d2 1.80 0.50 1 12 0.00 14.85
c20t1d2 96.41 0.97 494 16952 0.59 6.29
c20t1d4 96.81 3.19 1094 42291 2.56 5.38
c20t1d6 96.01 0.67 312 15246 0.39 8.05
c20t2d0 8.78 0.69 22 341 0.33 7.88
c20t4d0 4.39 0.50 15 367 0.00 3.09
14:28:58 c0t6d0 3.81 0.50 9 60 0.00 7.16
c2t6d0 1.40 0.50 6 47 0.00 3.54
c22t1d1 42.08 0.52 54 2125 0.11 13.33
c22t1d3 76.95 0.67 284 10344 0.28 8.21
c22t1d5 85.37 1.24 573 25161 1.21 6.44
c20t1d2 99.80 0.79 449 19562 0.63 8.47
c20t1d4 99.60 3.37 1312 73558 2.49 5.11
c20t1d6 100.00 0.68 393 16648 0.36 8.80
c20t2d0 12.02 0.50 39 584 0.00 3.26
c20t2d5 0.20 0.50 0 3 0.00 4.15
c20t4d0 0.40 0.50 0 6 0.00 6.90
14:29:03 c0t6d0 15.97 0.50 65 649 0.00 2.80
c2t6d0 3.59 0.50 42 407 0.00 1.16
c22t7d0 0.40 0.50 0 0 0.00 19.69
c22t1d1 26.35 0.69 53 3909 0.90 10.68
c22t1d3 80.84 0.68 193 6253 0.91 13.07
c22t1d5 71.26 0.86 436 10820 0.98 6.51
c22t2d1 1.60 0.50 0 3 0.00 40.25
c22t0d2 3.39 0.50 2 26 0.00 19.04
c20t6d7 0.20 0.50 0 0 0.00 1.60
c20t7d0 0.20 0.50 0 0 0.00 9.27
c20t1d2 95.41 0.86 518 36126 0.52 6.30
c20t1d4 97.41 3.37 1192 52532 2.59 5.06
c20t1d6 83.83 0.61 280 14202 0.25 9.02
c20t2d0 15.37 0.54 36 1460 0.49 13.39
c20t4d0 0.20 0.50 2 42 0.00 2.36
14:29:08 c0t6d0 5.00 0.50 13 95 0.00 5.87
c2t6d0 1.80 0.50 9 74 0.00 2.60
c22t1d1 42.20 0.53 91 5280 0.44 11.88
c22t1d3 99.80 0.68 329 10437 0.40 9.98
c22t1d5 52.00 0.94 477 14771 0.73 3.69
c22t0d2 0.20 0.50 0 4 0.00 4.33
c20t1d2 98.80 0.78 566 33069 0.56 7.08
c20t1d4 96.20 2.68 1258 61984 2.58 4.60
c20t1d6 70.00 0.62 335 17792 0.25 6.93
c20t2d0 4.60 0.50 12 200 0.03 7.48
c20t4d0 3.80 0.50 10 189 0.00 4.22
14:29:13 c0t6d0 4.61 0.50 13 227 0.00 5.62
c2t6d0 1.20 0.50 7 100 0.00 3.68
c22t1d1 53.31 0.55 115 3319 0.70 11.37
c22t1d3 100.00 0.73 314 8696 0.66 10.19
c22t1d5 39.88 0.53 448 13108 0.06 2.02
c20t1d2 100.00 2.31 1085 57267 2.06 5.90
c20t1d4 91.98 3.14 1156 44855 2.39 4.57
c20t1d6 65.33 0.57 331 18360 0.18 7.17
c20t2d0 3.01 0.50 10 149 0.00 5.41
c20t4d0 2.20 0.50 5 80 0.00 5.56
14:29:18 c0t6d0 4.80 0.50 19 121 0.00 3.38
c2t6d0 1.20 0.50 14 72 0.00 1.15
c22t1d1 95.00 0.75 333 8938 0.70 9.08
c22t1d3 71.60 0.77 232 7466 0.84 8.83
c22t1d5 63.00 1.08 510 14374 0.86 4.45
c22t0d2 0.40 0.50 1 8 0.00 2.69
c20t1d2 99.20 2.02 1226 56413 1.51 4.78
c20t1d4 99.80 3.17 1076 34579 1.80 4.67
c20t1d6 66.60 0.58 264 13546 0.21 8.60
c20t2d0 12.80 0.50 21 318 0.00 10.75
c20t4d0 3.00 0.50 4 115 0.00 7.48

Average c0t6d0 5.60 0.50 20 172 0.00 3.60
Average c2t6d0 1.37 0.50 13 101 0.00 1.45
Average c22t1d1 68.57 0.60 149 4534 0.44 10.75
Average c22t1d3 91.45 0.81 296 10011 0.66 10.19
Average c22t1d5 74.94 1.12 525 23054 1.05 5.87
Average c22t2d1 0.20 0.50 0 2 0.00 20.53
Average c22t0d2 0.90 0.50 1 9 0.00 10.32
Average c20t1d2 88.34 1.18 597 27810 0.80 5.40
Average c20t1d4 92.02 2.73 936 34858 2.32 5.42
Average c20t1d6 67.12 0.63 322 16927 0.27 6.56
Average c20t2d0 11.55 0.52 30 561 0.07 5.97
Average c20t4d0 2.10 0.50 7 148 0.00 3.12
Average c22t6d7 0.03 0.50 0 0 0.00 19.76
Average c20t4d1 0.02 0.50 0 0 0.00 13.60
Average c20t6d7 0.05 0.50 0 0 0.00 10.93
Average c20t7d0 0.03 0.50 0 0 0.00 14.13
Average c22t4d2 0.02 0.50 0 0 0.00 9.80
Average c20t4d2 0.03 0.50 0 0 0.00 12.79
Average c20t2d5 0.02 0.50 0 0 0.00 4.15
Average c22t7d0 0.03 0.50 0 0 0.00 19.69


What should I get out of this? What could be making it slow... Slow only for the users...

thanks
Jonathan

[skt_skt]

those service time does not look bad. Also not that those values you see are blended across metas(i mean it will have a higher value the disk invloved may be a comination of multile matas.; for a 64GB disk it will have a 8*8GB meta; one meta haead and other meta memmers)

But definitly you can take that as a trend of increasing IO acticities.

Do you have glance installed? Could you run "glance -f /tmp/glance.out",
select PID 561, let glance screen print, press "F" for process open
files and let it screen print, and finally, press W for wait states, let
it screen print. Please quit glance now and see the o/p file



see a bad IO trend here, 100% IO wait

ait States PID: 561, loadfile PPID: 1 euid: 0 User:
root

Event % Blocked On %
------------------------------------------------------------------------
--------
IPC : 0.0 Cache : 0.0
Job Control: 0.0 CDROM IO : 0.0
Message : 0.0 Disk IO : 0.0
Pipe : 0.0 Graphics : 0.0
RPC : 0.0 Inode : 0.0
Semaphore : 0.0 IO : 100.0
Sleep : 0.0 LAN : 0.0
Socket : 0.0 NFS : 0.0
Stream : 0.0 Priority : 0.0
Terminal : 0.0 System : 0.0
Other : 0.0 Virtual Mem: 0.0
C - cum/interval toggle % - pct/absolute toggle

[Tim Nelson]

Some disk queing with low service times.....

hmmm..

Might want to look at your queue_depth throttle

scsictl -m queue_depth $rdev

default is 8 most arrays can handle much more.

Review with Hitatchi to see what they say about it. Review this forum for other opinions as well.

[Jeeshan]

Hi Jonathan

Disk %busy output indicates that your disk has high I/O. Normally, if its near abt 5its value 50-60 % is normal. but in your case its worried. and also consider the value "avwait".

You may consider the follwoing steps to figure out your action plan.

1. though it is oracle and applications are used, ask DBA about the value of read and write value. If read is larger than write than you may decide to increase your readcache both in system and SAN. make also sure abt the disks raid group(1_0 or 5).

2. Check the kernel parameter of your systems buffer cache usage. If your buffer cache %write is lower than 20-30 you may need to change kernel parameters.