CPU Utilization

[Waqar Razi]

One of my customers has a complain about increased CPU utilization. I am a bit confused. When I do the top command, around 80% of CPU seems to be idle:

System: apple Fri Oct 2 13:37:26 2009
Load averages: 0.29, 0.29, 0.28
1105 processes: 1089 sleeping, 15 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.32 10.3% 0.0% 2.6% 87.1% 0.0% 0.0% 0.0% 0.0%
1 0.29 11.7% 0.0% 2.2% 86.1% 0.0% 0.0% 0.0% 0.0%
2 0.27 14.9% 0.0% 1.6% 83.5% 0.0% 0.0% 0.0% 0.0%
3 0.30 13.9% 0.0% 2.2% 83.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.29 12.7% 0.0% 2.2% 85.1% 0.0% 0.0% 0.0% 0.0%

I have also used the sar command. Its output shows that time utilized in waiting for I/O is around 70% and the idle time is much less.

13:54:20 %usr %sys %wio %idle
13:55:20 19 5 70 5
13:56:20 19 4 70 7
13:57:20 21 5 68 7
13:58:20 24 4 65 6
13:59:20 17 3 72 8
14:00:20 22 7 63 8
14:01:20 22 4 69 5
14:02:20 21 4 68 7
14:03:20 19 3 68 9
14:04:20 18 4 70 8
14:05:20 18 4 70 7

Average 20 4 68 7

Can some one please explain it to me if there is really some problem with the CPU utilization or this is caused by some slower I/O device or theire is no problem at all.

Any help will be highly appreciated.

[Michael Steele_2]

Hi

$WIO is high. These are processes waiting for I/O to complete. What are these reports?

UNIX95=1 ps -ef -o vsz,pid,ppid,comm | sort -rn | head -10

UNIX95=1 ps -ef -o pcpu,pid,ppid,comm | sort -rn | head -10

[Waqar Razi]

Here is the output of the commands mentioned by you:

UNIX95=1 ps -ef -o vsz,pid,ppid,comm | sort -rn | head -10
181964 2872 1 dsmc
62732 23718 1 PatrolAgent
33840 11595 1 mhragent
32640 3784 1 ia64_corehw
17272 1959 1 midaemon
12288 2005 1 scopeux
9652 1735 1 cimserver
9532 23851 23839 bgscollect
9268 1856 1 pdclientd
8820 3821 1 lpmc_em

UNIX95=1 ps -ef -o pcpu,pid,ppid,comm | sort -rn | head -10
12.52 20988 22949 mpmse
11.73 24410 22949 mpmse
6.34 2602 22949 mpmse
5.43 21110 22949 mpmse
4.29 3725 22949 mpmse
4.16 13599 22949 mpmse
2.46 21894 22949 mpmse
2.19 22311 22949 mpmse
2.17 20611 22949 mpmse
2.12 21543 22949 mpmse

In addition, I also ran the iostat command and got the following output:

iostat 4 4

device bps sps msps

c2t1d0 0 0.0 1.0
c2t0d0 0 0.0 1.0
c4t0d1 0 0.0 1.0
c4t0d2 0 0.0 1.0
c4t0d3 0 0.0 1.0
c4t0d4 0 0.0 1.0
c4t0d5 0 0.0 1.0
c4t0d6 0 0.0 1.0
c4t0d7 0 0.0 1.0
c4t1d0 0 0.0 1.0
c4t1d1 0 0.0 1.0
c4t1d2 0 0.0 1.0
c6t0d1 0 0.0 1.0
c6t0d2 0 0.0 1.0
c6t0d3 0 0.0 1.0
c6t0d4 0 0.0 1.0
c6t0d5 0 0.0 1.0
c6t0d6 0 0.0 1.0
c6t0d7 0 0.0 1.0
c6t1d0 0 0.0 1.0
c6t1d1 0 0.0 1.0
c6t1d2 0 0.0 1.0

c2t1d0 52 17.7 1.0
c2t0d0 43 10.1 1.0
c4t0d1 425 42.0 1.0
c4t0d2 451 47.7 1.0
c4t0d3 385 47.4 1.0
c4t0d4 519 57.5 1.0
c4t0d5 477 59.5 1.0
c4t0d6 417 56.8 1.0
c4t0d7 470 57.7 1.0
c4t1d0 0 0.0 1.0
c4t1d1 0 0.0 1.0
c4t1d2 0 0.0 1.0
c6t0d1 344 38.3 1.0
c6t0d2 353 41.8 1.0
c6t0d3 333 28.7 1.0
c6t0d4 278 36.1 1.0
c6t0d5 316 37.6 1.0
c6t0d6 356 47.2 1.0
c6t0d7 275 28.5 1.0
c6t1d0 0 0.0 1.0
c6t1d1 0 0.0 1.0
c6t1d2 0 0.0 1.0

c2t1d0 32 9.2 1.0
c2t0d0 30 8.5 1.0
c4t0d1 460 48.5 1.0
c4t0d2 557 65.8 1.0
c4t0d3 534 63.0 1.0
c4t0d4 506 57.8 1.0
c4t0d5 481 56.8 1.0
c4t0d6 441 48.8 1.0
c4t0d7 500 57.2 1.0
c4t1d0 0 0.0 1.0
c4t1d1 0 0.0 1.0
c4t1d2 0 0.0 1.0
c6t0d1 442 55.2 1.0
c6t0d2 508 51.5 1.0
c6t0d3 463 47.8 1.0
c6t0d4 393 53.8 1.0
c6t0d5 419 58.8 1.0
c6t0d6 388 48.5 1.0
c6t0d7 410 48.2 1.0
c6t1d0 0 0.0 1.0
c6t1d1 0 0.0 1.0
c6t1d2 0 0.0 1.0

c2t1d0 58 15.2 1.0
c2t0d0 42 10.2 1.0
c4t0d1 466 55.0 1.0
c4t0d2 465 58.2 1.0
c4t0d3 465 56.5 1.0
c4t0d4 519 58.7 1.0
c4t0d5 547 61.2 1.0
c4t0d6 532 67.2 1.0
c4t0d7 406 44.0 1.0
c4t1d0 0 0.0 1.0
c4t1d1 0 0.0 1.0
c4t1d2 0 0.0 1.0
c6t0d1 420 53.7 1.0
c6t0d2 476 60.7 1.0
c6t0d3 489 57.7 1.0
c6t0d4 494 67.4 1.0
c6t0d5 469 53.5 1.0
c6t0d6 452 57.0 1.0
c6t0d7 384 48.3 1.0
c6t1d0 0 0.0 1.0
c6t1d1 0 0.0 1.0
c6t1d2 0 0.0 1.0

[Tim Nelson]

first thought,

Need to look at this over time not just one snapshot. Review stats over time and find out where the issue really is.

Initially it looks like disk io but....

If you could post some sar info ( it is easier to read, at least for me ).
sar 10 10
sar -d 10 10

If the disk are extremely busy, there are queues, and CPU is low then you need to investigate the disk bottleneck, not a CPU issue.

( glance is a much better tool if you have it.)

once the real issue is determined the solutions will spawn from there.

[Waqar Razi]

sar -d 10 10

HP-UX m700004 B.11.11 U 9000/800 10/02/09

14:49:46 device %busy avque r+w/s blks/s avwait avserv
14:49:56 c2t1d0 5.38 0.54 12 100 0.23 7.17
c2t0d0 3.49 0.50 9 68 0.00 4.81
c4t0d1 28.02 0.50 34 584 0.00 8.98
c4t0d2 36.59 0.50 44 732 0.00 9.27
c4t0d3 32.40 0.50 44 749 0.00 8.29
c4t0d4 35.39 0.50 43 686 0.00 9.01
c4t0d5 41.48 0.50 47 751 0.00 10.20
c4t0d6 47.06 0.50 62 991 0.00 8.81
c4t0d7 40.58 0.50 45 727 0.00 10.12
c6t0d1 29.61 0.50 36 665 0.00 8.99
c6t0d2 34.00 0.50 44 737 0.00 8.78
c6t0d3 31.90 0.50 40 658 0.00 8.60
c6t0d4 35.09 0.50 43 683 0.00 8.97
c6t0d5 45.06 0.50 53 849 0.00 9.98
c6t0d6 47.86 0.50 61 975 0.00 9.22
c6t0d7 37.19 0.50 45 715 0.00 9.34
14:50:06 c2t1d0 15.50 1.83 48 341 3.46 8.54
c2t0d0 11.80 1.64 40 290 2.57 7.25
c4t0d1 31.90 0.50 42 790 0.00 8.60
c4t0d2 38.00 0.50 55 937 0.00 7.70
c4t0d3 32.80 0.50 53 908 0.00 6.83
c4t0d4 38.30 0.50 47 750 0.00 9.44
c4t0d5 46.20 0.50 59 949 0.00 9.24
c4t0d6 51.00 0.50 66 1050 0.00 9.22
c4t0d7 40.20 0.50 49 778 0.00 9.17
c6t0d1 31.80 0.50 41 757 0.00 8.39
c6t0d2 37.50 0.50 56 974 0.00 7.30
c6t0d3 33.40 0.50 50 877 0.00 7.28
c6t0d4 36.60 0.50 46 744 0.00 8.75
c6t0d5 47.50 0.50 57 918 0.00 9.84
c6t0d6 51.30 0.50 66 1048 0.00 9.29
c6t0d7 41.60 0.50 49 784 0.00 9.75
14:50:16 c2t1d0 11.21 1.74 33 275 3.62 11.77
c2t0d0 7.81 1.76 26 233 3.51 10.84
c4t0d1 35.84 0.50 37 642 0.00 11.06
c4t0d2 36.34 0.50 46 771 0.00 9.44
c4t0d3 31.43 0.50 42 698 0.00 8.45
c4t0d4 37.34 0.50 39 628 0.00 10.94
c4t0d5 37.34 0.50 45 719 0.00 9.60
c4t0d6 56.66 0.50 69 1099 0.00 9.79
c4t0d7 36.54 0.50 43 695 0.00 9.48
c6t0d1 33.93 0.50 36 641 0.00 10.51
c6t0d2 34.53 0.50 41 683 0.00 9.24
c6t0d3 31.53 0.50 41 689 0.00 8.41
c6t0d4 43.44 0.50 41 653 0.00 12.28
c6t0d5 39.54 0.50 48 766 0.00 9.64
c6t0d6 56.56 0.50 68 1095 0.00 9.88
c6t0d7 33.13 0.50 43 682 0.00 8.82
14:50:20 24 4 64 8
14:50:26 c2t1d0 5.81 0.88 17 136 1.36 11.87
c2t0d0 4.50 1.02 13 120 1.74 10.30
c4t0d1 29.43 0.50 37 692 0.00 8.98
c4t0d2 30.43 0.50 49 866 0.00 6.94
c4t0d3 31.23 0.50 48 848 0.00 7.11
c4t0d4 39.24 0.50 43 682 0.00 10.39
c4t0d5 37.94 0.50 46 738 0.00 9.76
c4t0d6 53.45 0.50 62 999 0.00 10.26
c4t0d7 30.03 0.50 42 669 0.00 8.11
c6t0d1 30.03 0.50 35 671 0.00 9.54
c6t0d2 30.73 0.50 51 903 0.00 6.81
c6t0d3 30.13 0.50 49 885 0.00 6.71
c6t0d4 35.84 0.50 40 647 0.00 10.33
c6t0d5 37.64 0.50 45 716 0.00 9.71
c6t0d6 54.25 0.50 63 1015 0.00 10.21
c6t0d7 33.23 0.50 42 676 0.00 8.90
14:50:36 c2t1d0 3.90 0.50 9 73 0.00 9.59
c2t0d0 2.50 0.50 7 65 0.00 6.47
c4t0d1 36.46 0.50 41 743 0.00 9.85
c4t0d2 32.07 0.50 53 949 0.00 6.83
c4t0d3 29.47 0.50 46 842 0.00 6.95
c4t0d4 41.46 0.50 42 676 0.00 11.33
c4t0d5 37.86 0.50 49 777 0.00 8.90
c4t0d6 51.35 0.50 64 1018 0.00 9.73
c4t0d7 33.87 0.50 44 703 0.00 8.93
c6t0d1 31.37 0.50 37 710 0.00 9.31
c6t0d2 30.67 0.50 52 912 0.00 6.69
c6t0d3 33.67 0.50 50 880 0.00 7.78
c6t0d4 40.56 0.50 43 691 0.00 10.85
c6t0d5 38.06 0.50 48 772 0.00 9.06
c6t0d6 52.15 0.50 63 1010 0.00 9.76
c6t0d7 33.37 0.50 43 689 0.00 8.74
14:50:46 c2t1d0 8.10 0.50 18 143 0.00 8.63
c2t0d0 5.10 0.50 13 118 0.00 6.02
c4t0d1 29.80 0.50 36 634 0.00 9.16
c4t0d2 31.50 0.50 49 853 0.00 6.95
c4t0d3 31.90 0.50 48 843 0.00 7.19
c4t0d4 37.00 0.50 42 667 0.00 10.49
c4t0d5 39.80 0.50 50 808 0.00 9.10
c4t0d6 46.20 0.50 59 949 0.00 9.21
c4t0d7 34.80 0.50 43 682 0.00 9.25
c6t0d1 28.00 0.50 35 608 0.00 8.79
c6t0d2 31.20 0.50 47 822 0.00 7.64
c6t0d3 29.80 0.50 46 794 0.00 7.22
c6t0d4 34.40 0.50 42 680 0.00 9.25
c6t0d5 40.70 0.50 51 821 0.00 9.40
c6t0d6 49.40 0.50 60 960 0.00 9.71
c6t0d7 33.70 0.50 42 670 0.00 9.33
14:50:56 c2t1d0 4.00 0.50 10 97 0.00 9.06
c2t0d0 2.70 0.50 7 78 0.00 6.38
c4t0d1 26.40 0.50 32 552 0.00 9.02
c4t0d2 34.10 0.50 45 766 0.00 8.66
c4t0d3 27.30 0.50 39 663 0.00 7.46
c4t0d4 33.90 0.50 40 632 0.00 9.41
c4t0d5 39.20 0.50 44 701 0.00 10.36
c4t0d6 45.50 0.50 58 928 0.00 9.07
c4t0d7 37.20 0.50 43 694 0.00 9.69
c6t0d1 28.30 0.50 33 587 0.00 9.15
c6t0d2 33.60 0.50 45 758 0.00 8.61
c6t0d3 32.20 0.50 43 738 0.00 8.24
c6t0d4 32.80 0.50 38 614 0.00 9.36
c6t0d5 38.50 0.50 43 694 0.00 10.13
c6t0d6 43.50 0.50 55 886 0.00 9.08
c6t0d7 34.50 0.50 41 658 0.00 9.22
14:51:06 c2t1d0 6.00 0.50 14 104 0.00 6.31
c2t0d0 3.80 0.50 10 71 0.00 4.69
c4t0d1 32.00 0.50 42 707 0.00 8.47
c4t0d2 30.30 0.50 43 710 0.00 7.70
c4t0d3 32.70 0.50 46 775 0.00 8.08
c4t0d4 38.50 0.50 43 683 0.00 10.14
c4t0d5 42.70 0.50 51 816 0.00 9.81
c4t0d6 43.80 0.50 58 928 0.00 8.87
c4t0d7 37.50 0.50 44 709 0.00 9.24
c6t0d1 31.60 0.50 39 707 0.00 9.10
c6t0d2 31.30 0.50 44 759 0.00 7.94
c6t0d3 30.50 0.50 44 732 0.00 7.84
c6t0d4 37.90 0.50 42 674 0.00 10.19
c6t0d5 41.30 0.50 50 805 0.00 9.68
c6t0d6 45.80 0.50 59 941 0.00 8.94
c6t0d7 37.00 0.50 47 755 0.00 8.81
14:51:16 c2t1d0 6.30 0.51 15 131 0.10 8.55
c2t0d0 3.90 0.50 10 93 0.04 6.63
c4t0d1 36.60 0.50 41 734 0.00 9.94
c4t0d2 34.80 0.50 47 779 0.00 8.31
c4t0d3 27.60 0.50 39 665 0.00 7.97
c4t0d4 32.00 0.50 42 664 0.00 8.51
c4t0d5 39.30 0.50 49 789 0.00 9.13
c4t0d6 41.40 0.50 55 880 0.00 8.53
c4t0d7 36.90 0.50 50 794 0.00 8.44
c6t0d1 31.90 0.50 39 710 0.00 9.07
c6t0d2 33.90 0.50 46 772 0.00 8.22
c6t0d3 28.60 0.50 40 671 0.00 7.91
c6t0d4 35.30 0.50 42 672 0.00 9.59
c6t0d5 41.80 0.50 50 795 0.00 9.40
c6t0d6 44.70 0.50 55 883 0.00 9.41
c6t0d7 34.60 0.50 45 715 0.00 8.75
14:51:20 28 4 59 9
14:51:26 c2t1d0 5.19 0.50 12 90 0.00 8.17
c2t0d0 3.80 0.50 9 79 0.00 5.91
c4t0d1 35.36 0.50 44 777 0.00 9.32
c4t0d2 31.97 0.50 50 855 0.00 7.14
c4t0d3 29.27 0.50 46 800 0.00 6.95
c4t0d4 36.26 0.50 46 732 0.00 8.98
c4t0d5 40.66 0.50 51 823 0.00 9.29
c4t0d6 37.46 0.50 48 766 0.00 9.20
c4t0d7 49.65 0.50 65 1037 0.00 8.96
c6t0d1 33.47 0.50 43 785 0.00 8.87
c6t0d2 33.27 0.50 53 907 0.00 7.28
c6t0d3 30.37 0.50 48 838 0.00 6.93
c6t0d4 39.96 0.50 48 775 0.00 9.44
c6t0d5 41.16 0.50 51 820 0.00 9.43
c6t0d6 35.86 0.50 44 705 0.00 9.16
c6t0d7 47.05 0.50 65 1039 0.00 8.39

Average c2t1d0 7.14 1.09 19 149 1.66 9.20
Average c2t0d0 4.94 1.10 14 121 1.51 7.53
Average c4t0d1 32.18 0.50 39 686 0.00 9.34
Average c4t0d2 33.61 0.50 48 822 0.00 7.85
Average c4t0d3 30.61 0.50 45 779 0.00 7.50
Average c4t0d4 36.94 0.50 43 680 0.00 9.85
Average c4t0d5 40.25 0.50 49 787 0.00 9.52
Average c4t0d6 47.39 0.50 60 961 0.00 9.29
Average c4t0d7 37.73 0.50 47 749 0.00 9.13
Average c6t0d1 31.00 0.50 37 684 0.00 9.16
Average c6t0d2 33.07 0.50 48 823 0.00 7.78
Average c6t0d3 31.21 0.50 45 776 0.00 7.65
Average c6t0d4 37.19 0.50 43 683 0.00 9.88
Average c6t0d5 41.13 0.50 50 796 0.00 9.63
Average c6t0d6 48.14 0.50 59 952 0.00 9.49
Average c6t0d7 36.54 0.50 46 738 0.00 8.98

[Tingli]

Don't see anything wrong with disk i/o. busy is less than 90, avque is less than 3 and avwait is much smaller than avserv. i/o should be fine.

[Michael Steele_2]

a) You have no disk bottlenecks. This is the sar -d output. Looks like your attached to a disk array that is moving data along very well. Not even your boot disks are io bound and boot disks always have the highest io.

b) Most of what is listed here are diagnostics related, midaemon, patrtlagent, scopeeux, cimserver. You can check for the latest diags patches but I doubt if this is the problem.

181964 2872 1 dsmc
62732 23718 1 PatrolAgent
33840 11595 1 mhragent
32640 3784 1 ia64_corehw
17272 1959 1 midaemon
12288 2005 1 scopeux
9652 1735 1 cimserver
9532 23851 23839 bgscollect
9268 1856 1 pdclientd
8820 3821 1 lpmc_em

c) You have several child processes spawned off from mpmse. This is consistent with the high $wio. In my opinion you should send this back to the application people.

UNIX95=1 ps -ef -o pcpu,pid,ppid,comm | sort -rn | head -10
12.52 20988 22949 mpmse
11.73 24410 22949 mpmse


d) Build a script with these commands and save the data. Add timestamps to the top of the script with comment bars for segregation of each report. Run the report every 20 minutes. Especially track the 'vsz' information. You are looking for a memory leak. For example, right now 181964 2872 1 dsmc has the highest 'vsz'. If it grows, then memory leak. However, this is / probably different from the $wio issues.

[Hein van den Heuvel]

>> One of my customers has a complain about increased CPU utilization.

Maybe the low idle time was interpreted as high utilization. That would be incorrect. The CPU is also idle during WIO.

Go back to the source.
Use any or all of the line below:

"How did you determine that the CPU increased?"
"What is the baseline used"
"All the time, or a specific time window"
"Please explain any problems, performance or otherwise, which you believe might be cause by high cpu utilization"..
"You bought it, now it is being used. Great. Tell your boss you gave the right advice"
"Don't worry, be happy"

Hope this helps,
Hein van den heuvel
HvdH Performance Consulting

[Hakki Aydin Ucar]

You do not have a -CPU Bottleneck as everyone said, especially because of low cpu utility.

-Disk bottleneck ;
Even though some of your disks in limit ( %busy >50 ) in sampled period you need to monitor some time according to experts a better question would be is it much higher than normal for your system ?

But on the other hand you don t have avwait > avserv for your disks ,so it is good for you.

you can also run "vmstat 5 30" would be better if a large run queue with idle CPU ?