high %usr low %idle

[Steve_3]

Below is the sar report...In block of times, %idle is < 5. Run queue avg ~2. This system have 4 cpu. Is there a cpu bottleneck during those times? Will adding more cpu help? Some users are complaining their process are slow during that time.

00:00:00 %usr %sys %wio %idle
01:00:01 37 13 20 30
02:00:00 17 4 12 67
03:00:00 8 7 42 44
04:00:00 38 10 42 10
05:00:00 50 7 13 31
06:00:00 36 6 10 48
07:00:01 42 7 9 42
08:00:01 46 8 12 34
08:20:01 49 9 23 19
08:40:01 42 9 16 33
09:00:01 50 8 10 32
09:20:01 76 10 9 5
09:40:01 56 10 19 15
10:00:00 70 9 8 13
10:20:00 67 10 8 14
10:40:00 63 9 8 19
11:00:00 77 12 4 7
11:20:00 78 11 6 6
11:40:00 69 10 8 13
12:00:01 67 8 7 18
12:20:01 47 9 7 38
12:40:01 66 13 7 14
13:00:00 73 16 8 2
13:20:00 82 17 1 0
13:40:00 80 15 5 0
14:00:01 70 11 12 7
14:20:01 89 11 0 0
14:40:01 85 13 2 1
15:00:01 85 12 2 1
15:20:01 85 13 1 0
15:40:01 84 14 2 0
16:00:00 82 11 4 3
16:20:00 71 9 9 11
16:40:00 88 10 2 0
17:00:00 87 9 2 1
17:20:00 86 8 4 2
17:40:00 63 8 16 13
18:00:00 49 6 10 35
19:00:00 40 5 6 48
20:00:01 12 3 3 82
21:00:00 7 3 3 87
22:00:00 14 4 10 72
23:00:00 30 4 26 40

Average 47 8 12 32

[Steven E. Protter]

What kind of war run was this?

Seems wio, wait io is where the issue is. I don't think adding a cpu is going to help so much.

SEP

[Florian Heigl (new acc)]

Actually You'd have to see the load of the individual CPUs - if there is a non-threaded application keeping one cpu at 100% all day, there might be a bottleneck. But that wouldn't be solveable by adding another cpu ;)

I think the load is okay - the 47% average sums up to 'two cpus could handle this load', so You are on the safe side, and the 70% peak means that even during maximum load You still have enough CPU power (20%) left.

But You should really have a closer look at cpu run queues etc. using glance. Real bottlenecks would show up there.
Also, You could add a sar -d over the same time period and look at the disk busy and avqueue values, maybe You can lower those 20% IO wait a bit.

[A. Clay Stephenson]

You have a few peak periods when the CPU's are fully utilized, so yes, you could benefit from more CPU throughput.

In general, fewer, faster CPU's beat more, slower CPU's because there is additional synchronization overhead with each additional CPU. The benefit gained from each additional CPU becomes less and less -- in economic terms, diminishing marginal utility.

The good news is that you have a high fraction of user (as opposed to system) CPU usage so that the box is actually doing useful work.

[Alex Lavrov.]

Try to install System Healthchek and let it run for couple of days, it will output nice HTML that will tell you your problems with CPU.

Another thing, ty running ps command to see wich processes take CPU.

export UNIX95=1 (for XPG4)
ps -ef -o cpu,args | sort -n | tail -n 10

that will give you top 10 cpu consumers.

System Healthcjeck can be found here:
http://software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=SHCBASE01

(there are also nice professional report, but it costs $)

[Tim D Fulford]

Hi

I think you are CPU bound; The "classic" is
high CPU activity > 70% (tick)
High priority queues >1 (need to check)
High run queues ~3

You can check the priority queues and run queue from MeasureWare / PerformanceAgent. If you do not have these.

see the HP-UX cookbook

http://h21007.www2.hp.com/dspp/ddl/ddl_Download_File_TRX/1,1249,280,00.pdf

Regards

Tim

[Tim D Fulford]

Despite being English born & bred I still cant write coherent sensible sentences.

If you do not have meauureware/OVPA you can install a trial version, alternatively the HP-UX cook book may give you some extra pointers...

Regards

Tim

[rick jones]

I'd like to reinforce the "check each CPU individually" suggestion...

One "benefit" of the CPU utilization being in user space is you have a better prospect of addressing it. One way might be to take some profiles with Prospect:

http://www.hp.com/go/prospect

which can tell you where each application is spending its CPU time. Even if you do not compile those applications yourself, you may be able to provide the data to the entities that do and get better versions.

For those applications you compile yourself, you might consider higher optimization levels with the compiler. I suspect there are writeups about such things somewhere on www.hp.com or devresource.hp.com - one other place you can go to look for ideas would be www.spec.org and look at the SPECcpu2000 results - in particular the compiler options for any and all components that look like they may be doing things similar to your applications. Between higher optimization levels, and small code tweaks one can sometimes get rather substantial performance improvements. One example, perhaps a bit dated, but still relevant at least for mothodology would be:

ftp://ftp.cup.hp.com/dist/networking/briefs/named_performance.txt