CPU IO always high

[Jack Fan]

I run BaaN III and ORACLE 7.3.4.5 onto HP-K580 (4 cpu, 4gb memory, AutoRAID) and around 150 users. Now, I face one problem that CPU utilization alway high and run queue exceed 3 or 4. What should I do?

For further information , please see the attachment!

Thank In Advance!!

Jack Fan

[Steven Sim Kok Leong]

Hi,

First, you need to check using sar, glance or top to identify the CPU bottleneck. Is there a high %wio in sar -u (indicates potential I/O bottleneck)? What are the top CPU hogs in the top output. Look at the glance charts. Are there any bottlenecks elsewhere such as pageouts or disk I/O bottlenecks

If it is confirmed that insufficient CPU resources is the cause for the CPU load problem, the usual solution is to finetune your sql statements. The common workaround is to add more CPUs.

Hope this helps. Regards.

Steven Sim Kok Leong

[Jack Fan]

Hi,

I ran sar -u 10 10 to gather cpu utilization data as following:

aip[root]:/:>sar -u 10 10

HP-UX aip B.11.00 A 9000/800 01/30/02

18:49:02 %usr %sys %wio %idle
18:49:12 80 17 3 0
18:49:22 80 15 5 0
18:49:32 77 16 7 0
18:49:42 78 16 6 0
18:49:52 81 15 3 0
18:50:02 79 16 5 0
18:50:12 79 17 4 0
18:50:22 80 14 6 0
18:50:32 81 14 5 0
18:50:42 81 14 4 0

Average 80 15 5 0

Is it high about %wio?

Jack Fan

[Steven Sim Kok Leong]

Hi,

Personally, I find the %wio within acceptable limits. However, notice that most of your CPU time is spent in %usr (averages around 80%) which refers to your application.

Show us your output for sar -w and sar -q.

# sar -w 1 10
# sar -q 1 10

Can you also give us your top output?

# top -d 1

Btw, do you have glance installed?

Hope this helps. Regards.

Steven Sim Kok Leong

[Jack Fan]

sar -w 1 10
HP-UX aip B.11.00 A 9000/800 01/30/02

19:05:13 swpin/s bswin/s swpot/s bswot/s pswch/s
19:05:14 0.00 0.0 0.00 0.0 2816
19:05:15 0.00 0.0 0.00 0.0 2908
19:05:16 0.00 0.0 0.00 0.0 1965
19:05:17 0.00 0.0 0.00 0.0 1704
19:05:18 0.00 0.0 0.00 0.0 2355
19:05:19 0.00 0.0 0.00 0.0 2364
19:05:20 0.00 0.0 0.00 0.0 2515
19:05:21 0.00 0.0 0.00 0.0 1636
19:05:22 0.00 0.0 0.00 0.0 2089
19:05:23 0.00 0.0 0.00 0.0 2839

Average 0.00 0.0 0.00 0.0 2319


sar -q 1 10
HP-UX aip B.11.00 A 9000/800 01/30/02

19:05:55 runq-sz %runocc swpq-sz %swpocc
19:05:56 1.0 25 0.0 0
19:05:57 1.5 51 0.0 0
19:05:58 3.0 25 0.0 0
19:05:59 1.0 50 0.0 0
19:06:00 3.0 25 0.0 0
19:06:01 3.0 51 0.0 0
19:06:02 2.0 25 0.0 0
19:06:03 1.5 50 0.0 0
19:06:04 2.5 50 0.0 0
19:06:05 3.0 25 0.0 0

Average 2.1 38 0.0 0


Load averages: 1.49, 1.58, 1.85
1208 processes: 1198 sleeping, 7 running, 3 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.09 51.5% 0.0% 48.5% 0.0% 0.0% 0.0% 0.0% 0.0%
1 1.83 96.0% 0.0% 2.0% 2.0% 0.0% 0.0% 0.0% 0.0%
2 1.51 9.0% 0.0% 10.0% 81.0% 0.0% 0.0% 0.0% 0.0%
3 1.54 98.0% 0.0% 1.0% 1.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.49 64.0% 0.0% 15.0% 21.0% 0.0% 0.0% 0.0% 0.0%

Memory: 1781688K (457600K) real, 1824292K (524572K) virtual, 761484K free Page#
1/135

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 16088 pb956069 241 20 11204K 9804K run 237:16 93.07 92.91 bshell
3 ? 17336 np220001 241 20 10152K 760K run 12:18 83.03 82.89 oracleAIPBA
1 ? 17098 np220001 241 20 10184K 792K run 14:41 81.16 81.01 oracleAIPBA
3 ? 12152 nw988133 154 20 10472K 1080K sleep 4:23 9.52 9.50 oracleAIPBA
1 ? 14579 nw001010 154 20 10312K 920K sleep 2:37 7.37 7.35 oracleAIPBA
3 ? 7665 nw988103 154 20 10552K 1160K sleep 5:14 4.87 4.86 oracleAIPBA
3 ? 4646 nm996199 154 20 10664K 1272K sleep 1:20 4.46 4.45 oracleAIPBA

[Krishna Prasad]

Did you know what your Oracle hit ratio is for your SGA? Or what your Oracle SGA is set to?

As mentioned in an earlier reply, output showing high % of user times points me to the application.

[Bill Hassell]

It's important to note that 100% CPU (that is, uptime loads of 4) are just fine, and in many cases, you can see loads that are 6 or 8. It all depends on the applications. If you stop running the apps, you'll see that the loads go to zero.

So a load factor of 3-4 is just fine. Now if the applications seem to run slow, there isn't much you can do in HP-UX to fix a slow application. Your version of Oracle is very old and inefficient SQL statements can very easily cause unnecessary CPU usage. I would suggest running Oracle stats on the SQL statements to see what overhead each requires and which are called the most often.

Also, Oracle can run siginificantly faster if there is enough SGA to perform sorts in RAM versus temp areas on disk. If you're running 10.20, you will be heavily constrained by the 3.75 Gb RAM limit. The only way to fix this is to upgrade HP-UX, Oracle (and probably Baan) to 11.0 or 11i 64bit mode for both the kernel as well as Oracle. Then you will no longer have any limits on memory addresses. Note that you'll need to plan for a minimum of 4Gb RAM, with better performance seen at 6-12 Gb or RAM.

[Krishna Prasad]

You may also want to run statistics on your Oracle tables. Oracle will make better use of indexes after statistics doing so. Perhaps this has not been ran in a long time.

[Steven Sim Kok Leong]

Hi,

From sar, your average run queue is 2.1 and % occupied is 38%. All your CPUs are busy, there is no CPU(s) that is idling. 0% idle is normal. In fact, that means you have paid your money's worth for your CPUs because they have not been wasting time idling! :-). Your deactivation rate is 0. These are pretty normal symptoms of a system, unless there has been a sudden increase given exact same conditions when the response time was faster (eg. no. of currently running) processes.

However, I would like to highlight the following:

1) process switching rate is 2319/sec
2) no. of concurrent processes is 1208.
3) 80% of CPU time is spent at the application.

These are all indicative of very busy application(s) running on your system.

In addition, what and why is this bshell program taking up 93% of all your overall CPU time?

0 ? 16088 pb956069 241 20 11204K 9804K run 237:16 93.07 92.91 bshell

You would want to check this process and program out if it was meant to run for 237:16 mins and consume such a big percentage of your CPU time.

Hope this helps. Regards.

Steven Sim Kok Leong

[Terry Johnson_1]

Jack,

We run BaanIV on a K580 with 6-way and 4 Gig of Ram. I don't know the differnce between Baan III and BaanIV, but the process "bshell" is what a user opens up when logging on then end when user logs off. This one process on your machine "bshell" has been running awhile. What is this?

Anyway,
If you have a $BSE/lib/defaults/db_resource
file there's a parameter called bdb_max_session_schedule:? which allows users to open multi-connections to the database. If you set this to 1 (this is what baan support recommended for us) this could relieve some contraints.

Good Luck,
Buzz