how to tune HP-UX Performance

[wangmx]

Hi,
Today I monitor by top our a database server,HP RP8420.as below:
System: host01 Tue Jun 7 23:04:41 2005
Load averages: 1.95, 1.90, 1.89
623 processes: 579 sleeping, 43 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 2.03 60.9% 0.0% 38.1% 1.0% 0.0% 0.0% 0.0% 0.0%
1 2.34 63.1% 0.0% 34.4% 2.6% 0.0% 0.0% 0.0% 0.0%
2 1.63 49.3% 0.0% 50.5% 0.2% 0.0% 0.0% 0.0% 0.0%
3 2.16 40.9% 0.0% 57.4% 1.8% 0.0% 0.0% 0.0% 0.0%
4 1.71 43.0% 0.0% 54.4% 2.6% 0.0% 0.0% 0.0% 0.0%
5 2.13 53.0% 0.0% 45.0% 2.0% 0.0% 0.0% 0.0% 0.0%
6 2.26 52.3% 0.0% 47.0% 0.8% 0.0% 0.0% 0.0% 0.0%
7 2.01 49.3% 0.0% 47.7% 2.9% 0.0% 0.0% 0.0% 0.0%
8 2.30 47.2% 0.0% 50.1% 2.8% 0.0% 0.0% 0.0% 0.0%
9 1.83 44.4% 0.0% 52.5% 3.1% 0.0% 0.0% 0.0% 0.0%
10 1.50 44.6% 0.0% 54.0% 1.4% 0.0% 0.0% 0.0% 0.0%
11 2.24 44.0% 0.0% 55.0% 1.0% 0.0% 0.0% 0.0% 0.0%
12 2.07 59.1% 0.0% 40.9% 0.0% 0.0% 0.0% 0.0% 0.0%
13 1.83 49.1% 0.0% 50.3% 0.6% 0.0% 0.0% 0.0% 0.0%
14 1.99 63.5% 0.0% 34.4% 2.2% 0.0% 0.0% 0.0% 0.0%
15 1.73 93.1% 0.0% 6.9% 0.0% 0.0% 0.0% 0.0% 0.0%
16 1.69 74.7% 0.0% 23.8% 1.6% 0.0% 0.0% 0.0% 0.0%
17 1.71 46.4% 0.0% 52.7% 1.0% 0.0% 0.0% 0.0% 0.0%
18 2.00 41.7% 0.0% 54.4% 3.9% 0.0% 0.0% 0.0% 0.0%
19 1.82 46.2% 0.0% 52.8% 1.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.95 53.2% 0.0% 45.2% 1.6% 0.0% 0.0% 0.0% 0.0%

from top ouput CPU IDEL is very low,and SYS is high.why? how to tune it?
thanks

[Tomek Gryszkiewicz]

Look at the memory and swap usage: maybe this is a bottleneck

-Tomek

[HGN]

Hi

Refer to this thread

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=852937

Rgds

HGN

[Geoff Wild · Accepted solution]

Wow - 20 CPU's - yet SYS is high - that is not good....That means the system is spending an excessive amount of time executing code within the kernel...

System CPU Bottlenecks

System CPU Bottleneck Recipe Ingredients:

CPU bottleneck symptoms from above, and
Most of the time spent in the kernel (GBL_CPU_SYS_MODE_UTIL > 50%).
If you are spending most of your CPU time in System mode, then you'll want to break that down further and see what activity is causing processes to spend so much time in the kernel. First, check to see if most of the overhead is due to context switching. This is the kernel running different processes all the time. If you're doing a lot of context switching, then you'll want to figure out why, because this is not productive work. This is a whole topic in itself, so jump down to the next section on Context Switching Bottlenecks. Assuming it isn't that, see if GBL_CPU_INTERRUPT_UTIL is > 30 percent. If so, you likely have some kind of I/O bottleneck instead of a CPU bottleneck (that is, your CPU bottleneck is being caused by an I/O bottleneck), or just maybe you have a flaky I/O card. Switch gears and address the I/O issue first (Disk or Networking bottleneck). Memory bottlenecks can also come disguised as System CPU bottlenecks: if memory is fully utilized and you see paging, look at the memory issue first.

Assuming at this point that most of your kernel time is spent in system calls (GBL_CPU_SYSCALL_UTIL >30%), it's time to try to see which specific system calls are going on. It's best if you can use Glance on the system at the time the problem is active. If you can do this, count your lucky stars and skip to the next paragraph. If you are stuck with looking at historical data or using other tools, it won't include specific system call breakdowns, so you'll need to try to work from other metrics. Try looking at process data during the bad time and see which processes are the worst (highest PROC_CPU_SYSCALL_UTIL) and look at their other metrics or known behavior to see if you can determine the reason why that process would be doing excessive syscalls.

If you can catch the problem live, you can use Glance to drill down further. We like to use gpm for this because of its more flexible sorting and metric selection. Go into Reports->System Info->System Calls, and in this window configure the sort field to be the syscall rate. The most-often called syscall will be listed first. You can also sort by CPU time to see which syscalls are taking the most CPU time, as some syscalls are significantly more expensive than others are. In gpm's Process List report, you can choose the PROC_CPU_SYS_MODE_UTIL metric to sort on and the processes spending the most time in the kernel will be listed first. Select a process from the list and pull down the Process System Calls report and (after a few update intervals) you'll see the syscalls that process is using. Keep in mind that not all system calls map directly to libc interfaces, so you may need to be a little kernel-savvy to translate syscall info back into program source code. Once you find out which processes are involved in the bottleneck, and what they are doing, the tricky part is determining why. We leave this as an exercise for the user!

Common programming mistakes such as repetitive gettimeofday() or select() calls (we've seen thousands per second in some poorly designed programs) may be at the root of a System CPU bottleneck. Another common cause is excessive stat-type file system syscalls (the find command is good at generating these, as well as shells with excessive search PATH variables). Once, we traced the root cause of a bottleneck back to a program that was opening and closing /dev/null in a loop!

On busy and large multiprocessor systems, system CPU bottlenecks can be the result of contention over internal kernel resources such as data structures that can only be accessed on behalf of one CPU at a time. You may have heard of "spinlocks," which is what happens when processors must sit and spin waiting for a lock to be released on things like virtual memory or I/O control structures. This shows up in the tools as System CPU time, and it's hard to distinguish from other issues. Typically, this is OK because there's not much from the sysadmin perspective that you can do about it anyway. Spinlocks are an efficient way to keep processors from tromping over critical kernel structures, but some workloads (like those doing a lot of file manipulations) tend to have more contention. If programs never make system calls, then they won't be slowed down by the kernel. Unfortunately, this is not always possible!

Here's a plug for a contrib system trace utility put together by a very good friend of ours at HP. It's called tusc, and it's very useful for tracing activity and system calls made by specific processes: very useful for application developers. It's currently available via the HP Networking Contrib Archive (see References section at the end of this article) under the misc directory. We would be remiss if we did not say that applications have been written that perform an enormous number of system calls and there is not much that we can do about it, especially if the application is a third-party application. We have also seen developers "choose" the wrong calls for performance. It's a complex topic that Stephen is prepared to go into at length over a beer.

Rgds...Geoff

[D Block 2]

There is more SYS/time compared to USER/time, maybe due to heavy system calls such as I/O. Here's a comparision from my system:

Also, what is your last line of top ? free memory is ok ?


System: mysystem Tue Jun 7 11:29:15 2005
Load averages: 1.09, 1.03, 0.97
441 processes: 395 sleeping, 46 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.96 50.6% 0.0% 11.3% 38.1% 0.0% 0.0% 0.0% 0.0%
1 0.99 52.8% 0.0% 14.3% 32.9% 0.0% 0.0% 0.0% 0.0%
2 0.86 50.0% 0.0% 17.5% 32.5% 0.0% 0.0% 0.0% 0.0%
3 1.10 50.4% 0.0% 15.5% 34.1% 0.0% 0.0% 0.0% 0.0%
4 1.21 60.1% 0.0% 15.3% 24.6% 0.0% 0.0% 0.0% 0.0%
5 0.96 53.8% 0.0% 12.9% 33.3% 0.0% 0.0% 0.0% 0.0%
6 0.96 53.0% 0.0% 11.7% 35.3% 0.0% 0.0% 0.0% 0.0%
7 1.01 52.4% 0.0% 12.3% 35.3% 0.0% 0.0% 0.0% 0.0%
8 0.94 53.2% 0.0% 12.7% 34.1% 0.0% 0.0% 0.0% 0.0%
9 0.93 47.2% 0.0% 12.9% 39.9% 0.0% 0.0% 0.0% 0.0%
10 0.92 52.8% 0.0% 12.5% 34.7% 0.0% 0.0% 0.0% 0.0%
11 1.04 55.2% 0.0% 13.3% 31.5% 0.0% 0.0% 0.0% 0.0%
12 0.91 54.2% 0.0% 13.3% 32.5% 0.0% 0.0% 0.0% 0.0%
13 0.85 55.0% 0.0% 13.3% 31.7% 0.0% 0.0% 0.0% 0.0%
15 0.89 54.8% 0.0% 11.9% 33.3% 0.0% 0.0% 0.0% 0.0%
16 1.19 57.7% 0.0% 15.7% 26.6% 0.0% 0.0% 0.0% 0.0%
17 1.23 68.7% 0.0% 14.1% 17.3% 0.0% 0.0% 0.0% 0.0%
19 1.38 61.3% 0.0% 18.7% 20.0% 0.0% 0.0% 0.0% 0.0%
20 1.28 66.3% 0.0% 17.9% 15.9% 0.0% 0.0% 0.0% 0.0%
21 0.98 64.7% 0.0% 17.9% 17.5% 0.0% 0.0% 0.0% 0.0%
23 1.50 64.7% 0.0% 16.3% 19.0% 0.0% 0.0% 0.0% 0.0%
24 1.20 61.3% 0.0% 15.3% 23.4% 0.0% 0.0% 0.0% 0.0%
25 1.35 62.3% 0.0% 17.9% 19.8% 0.0% 0.0% 0.0% 0.0%
27 1.47 65.7% 0.0% 15.9% 18.5% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.09 57.0% 0.0% 14.7% 28.3% 0.0% 0.0% 0.0% 0.0%

Memory: 10901448K (10722860K) real, 12343896K (12114216K) virtual, 8767848K free
Page# 1/20