cpu idle time too high.

[khalid72]

I have posted earlier regarding cpu idle time too high and seem like %wio is also high. the do while loop does drop the cpu idle time by 25% but that does not solve the original problem. after reading various articles online etc,, i learned that perhaps setting the dbc_max_pct will force the cpu to work instead of sitting idle. my current dbc_max_pct is 30% of 4 GB cache, which i think is high. People have suggested online that for 4GB cache it should not be more 600 mb. Some have suggested to bring it down to dbc_max_pct to 7 and dbc_min_pct to 5 or so.

how do you make the cpu not sit idle when instead of having bottleneck at I/O?

I am also attaching a word document of info below incase you are comfortable reading the info pasted below. Thanks for all your help in advance.


System: HP Thu Sep 14 15:47:46 2006
Load averages: 0.07, 0.07, 0.08
470 processes: 447 sleeping, 22 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.07 1.8% 0.0% 4.4% 93.8% 0.0% 0.0% 0.0% 0.0%
1 0.07 3.4% 0.0% 2.0% 94.6% 0.0% 0.0% 0.0% 0.0%
2 0.08 2.6% 0.0% 2.2% 95.2% 0.0% 0.0% 0.0% 0.0%
3 0.07 2.8% 0.0% 2.2% 95.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.07 2.8% 0.0% 2.6% 94.7% 0.0% 0.0% 0.0% 0.0%

Memory: 1916808K (858468K) real, 2932116K (1630316K) virtual, 3237024K free Page# 1/19

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND



*********************************************************
sar -b 1 100

HP-UX B.11.23 U ia64 09/14/06

15:48:28 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
15:48:29 0 76 100 32 24 0 134 134
15:48:30 0 324 100 95 181 47 139 138
15:48:31 1 55 98 14 63 78 124 122
15:48:32 2 21 90 36 26 0 140 123
15:48:33 0 28 100 17 36 53 145 125
15:48:34 0 240 100 33 41 20 148 130
15:48:35 0 32 100 23 37 38 135 136
15:48:36 0 86 100 18 24 25 137 135
15:48:37 0 21 100 23 22 0 138 134

**********************************************************
sar -d 1 100

HP-UX B.11.23 U ia64 09/14/06

15:48:43 device %busy avque r+w/s blks/s avwait avserv
15:48:44 c2t1d0 99.01 0.51 283 12709 0.08 11.03
15:48:45 c2t1d0 100.00 0.50 319 14758 0.00 11.79
15:48:46 c2t1d0 100.00 0.50 311 14044 0.11 13.11
15:48:47 c2t1d0 100.00 0.50 308 14856 0.00 10.53
15:48:48 c2t1d0 100.00 0.50 317 15764 0.06 14.06
15:48:49 c2t1d0 100.00 0.50 294 12482 0.00 13.10
15:48:50 c2t1d0 100.00 0.50 315 16090 0.00 13.43

******************************************************
sar -u 1 100

HP-UX B.11.23 U ia64 09/14/06

15:49:00 %usr %sys %wio %idle
15:49:01 2 4 58 36
15:49:02 4 2 68 26
15:49:03 2 3 60 35
15:49:04 2 4 61 33
15:49:05 2 2 59 37
15:49:06 2 3 51 44

*******************************************************
kcusage
Tunable Usage / Setting
=============================================
dbc_max_pct 23 / 30
maxdsiz 12521472 / 1073741824
maxdsiz_64bit 12582912 / 4294967296
maxfiles_lim 71 / 4096
maxssiz 1097728 / 8388608
maxssiz_64bit 524288 / 268435456
maxtsiz 32768 / 100663296
maxtsiz_64bit 117440512 / 1073741824
maxuprc 329 / 400
max_thread_proc 37 / 256
maxvgs 1 / 10
msgmni 293 / 2048
msgseg 0 / 16384
msgtql 0 / 6644
nfile 1930 / 65536
nflocks 19 / 4096
ninode 1472 / 4880
nkthread 646 / 8416
nproc 472 / 4200
npty 1 / 60
nstrpty 16 / 60
nstrtel 0 / 60
nswapdev 1 / 10
nswapfs 0 / 10
semmni 30 / 2048
semmns 432 / 4096
shmmax 202334208 / 1073741824
shmmni 7 / 400
shmseg 654 / 300

********************************************************

swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 2275 -2275
memory 8180 1583 6597 19%
total 12276 3858 8418 31% - 0 -

*********************************************************

ipcs -moba
IPC status from /dev/kmem as of Thu Sep 14 15:54:04 2006
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c0188 --rw-rw-rw- root root root root 0 348 708 708 14:45:30 14:45:30 14:45:24
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 708 708 14:45:26 14:45:30 14:45:24
m 2 0x4120405e --rw-rw-rw- root root root root 1 8192 708 720 14:45:26 14:45:24 14:45:24
m 2097155 0xd62634a4 --rw-rw---- oracle dba oracle dba 56 202334208 23903 1892 14:50:27 15:47:05 10:03:36
m 65540 0x464d0000 --rw-rw-rw- root kusr root kusr 227 1568 4197 25007 15:53:04 15:53:04 15:13:30
m 229381 0x464a0000 --rw-rw-r-- root kusr root kusr 226 16777216 18473 25007 15:53:04 no-entry 10:50:02
m 393222 0x464a0001 --rw-rw-r-- knsw kusr knsw kusr 137 16777216 18726 28284 15:54:04 no-entry 10:58:29

*********************************************************
vmstat 4 4
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 8 0 394330 807455 9 1 1 0 0 0 2 1859 8849 673 1 1 97
2 8 0 394330 807358 9 0 0 0 0 0 0 3344 15162 1765 2 3 95
2 8 0 394330 807358 10 0 0 0 0 0 0 3325 15062 1716 2 3 95
4 6 0 394022 807199 13 0 0 0 0 0 0 3281 14607 1697 3 2 95

[spex · Accepted solution]

Hi,

The CPUs are sitting idle because it's blocking on disk I/O. Just look at the high %wio. Forcing the CPUs to work harder won't improve throughput.

'sar -b 1 100' shows %rcache at nearly 100%, which means that your buffer cache size should be increased, not decreased. The fact that your dynamic buffer cache has consumed all available memory but %rcache is pegged implies you need additional free memory. Either buy more RAM or reduce the amount used by your applications. Also take a look at how Oracle's SGA is configured.

Why does 'sar -d 1 100' show but a single device? If you are using an array, why is there only one LUN? Is it configured for raw I/O?

What type of storage do you have, and how is it attached to your server? What are users doing when the I/O bottleneck arises?

PCS

[Bill Hassell]

To increase CPU usage in a typical database system, you must reduce I/O and the only way you can do this is to put most of the disk records in memory. You need to double or triple the amount of RAM you have, then increase the dynamic buffer cache to about 1200 megs and tell your DBAs to use more RAM for shared memory. Of course, this assumes your DBAs know about tuning the database to use memory for full row insertions, to sort temp files and to buffer I/O.

[Prasanth B]

Hi,

I agree with your understanding of dbc_max_pct. You will need to drop down dbc_max_pct to 10% and dbc_max_pct to 5%. I have worked with pretty large oracle DBs and 10% max is more than sufficient. Quickly going through the stats, you have only 4GB of disk swap. I would recommend a minimum of 8GB (or more)

-PB

[spex]

Prasanth,

The system does not need more swap. Just look at the stats--'swapinfo -tm' shows 'dev used' at 0% and vmstat shows 'po' at 0. No paging is occurring.

Khalid first needs to increase available RAM. If he accomplishes this by adding additional memory, then he may be able to lower dbc_max_pct. Since dbc_max_pct is a percentage of overall RAM, if the memory pool grows, dbc_max_pct can be decreased as there is more memory to work with.

Of course, a static OS buffer cache is almost always preferable to a dynamic one. But in either case, supply must be greater than demand.

PCS

[Marvin Strong]

Based on what you show here. I'm in the increase memory camp also.

Grab some stats on current usage. For comparison after action is taken.

1 Increase Memory (at least double it, as suggest above)
2 Have DBA's tune database

Compare new stats with old checking to see where any additional tuning may help performance further. This might be kernel parameters or further tuning within the database. Database tuning almost always has better gains than system tuning.

[khalid72]

Thanks every one. I will try with increasing more ram and see what difference does it make. I assume not to decrease dbc_max_pct for now.