Load is very High ,But CPU utli is low comparing to load

[zs_hzh]

is anyone there can help me to find out where is the bottleneck of my server ?
2*cpu 4G Mem

load always >3 ,runing job in queues >10 ,this indicate me to adding CPU for processing all jobs??

but the cpu usage is not very high ,around 20%
Could you tell me what's go on with my system?

is all process waiting for memory ,so the total time in cpu is long ,but the phyiscal execution time is short?

what should i do right now to detemine the actual reason of this situation?

tks you all for your response

System: nbnkscs1 Tue Nov 22 17:01:03 2005
Load averages: 5.17, 5.57, 5.28
458 processes: 346 sleeping, 112 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 4.70 10.8% 7.1% 13.6% 68.5% 0.0% 0.0% 0.0% 0.0%
1 5.65 7.7% 2.0% 12.2% 78.2% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 5.17 9.3% 4.5% 12.8% 73.4% 0.0% 0.0% 0.0% 0.0%


nbnkscs1:/ (105) root% vmstat 2 10
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
15 0 0 157598 175023 136 23 0 2 38 0 0 900 9535 1710 22 9 69
10 0 0 160258 176878 185 49 2 1 29 0 0 918 11234 2257 85 12 3
10 0 0 160258 171182 217 38 1 1 24 0 0 821 11780 2231 86 13 1
10 0 0 191845 169618 154 33 1 0 24 0 0 840 10128 2228 63 13 24
10 0 0 191845 170694 128 24 0 4 15 0 0 857 9209 1953 31 6 62
10 0 0 191845 169569 166 15 0 5 10 0 0 960 11120 2266 38 17 45
14 0 0 193666 169567 147 16 0 3 12 0 0 861 10430 2109 24 7 69
14 0 0 193666 170552 145 17 0 2 14 0 0 842 9587 1912 16 12 72
2 0 0 151653 170035 147 14 0 1 12 0 0 782 9252 1705 9 11 80
2 0 0 151653 170034 136 8 0 1 7 0 0 825 9027 1712 25 8 68


------------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU U | 15% 30% 53%
Disk Util F F | 19% 23% 36%
Mem Util S SU UB B | 86% 86% 86%
Swap Util U UR R | 48% 48% 48%
------------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 14
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 200% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
s4ltrfmx 9502 5874 154 omc 0.0/ 0.1 0.0 0.0/ 0.0 167.5mb 9
java 2102 2100 168 root 0.0/ 0.2 0.1 0.0/ 0.0 128.6mb 11
jav 6197 5942 168 omc 0.0/ 0.2 0.1 0.0/ 0.0 115.1mb 101
PA_ 6992 1 168 omc 0.6/ 0.9 0.3 0.0/ 0.0 112.4mb 68
java 1575 1 168 root 0.8/ 2.1 0.7 0.0/ 0.1 89.6mb 91
cpmmanmx 11755 1 154 wangguan 0.6/ 8.1 2.7 0.0/ 0.0 57.4mb 1
zrnevhmx 9133 5874 154 omc 0.0/ 0.0 0.0 0.0/ 0.0 51.8mb 1
native_thre 8961 8953 168 omc 0.2/ 0.5 0.2 0.0/ 0.0 49.1mb 25
native_thre 10208 10206 168 omc 0.0/ 1.4 0.5 0.0/ 0.0 45.6mb 24
native_thre 8891 7326 168 omc 0.0/ 0.1 0.0 0.0/ 0.0 38.5mb 37
native_thre 9349 9342 168 omc 0.0/ 0.2 0.1 0.0/ 0.0 36.9mb 27
deiparmx 14980 6093 154 omc 1.5/ 1.8 0.6 0.0/ 0.0 30.9mb 2
kapnssmx 7619 5874 168 omc 0.0/ 0.0 0.0 0.0/ 0.0 28.7mb 1
native_thre 9159 9147 168 omc 0.0/ 0.4 0.1 0.0/ 0.0 26.4mb 16
guimanmx2 16010 1 154 omc 0.0/ 0.1 0.0 0.0/ 0.0 26.0mb 2
guimanmx2 16209 1 154 omc 0.0/ 0.1 0.0 0.0/ 0.0 26.0mb 2
guimanmx2 11007 1 154 wangguan 0.0/ 0.1 0.0 0.0/ 0.0 24.5mb 2
s4lalfmx 13688 5874 168 omc 0.0/ 0.0 0.0 0.0/ 0.0 23.6mb 9
arcq3cmx 7623 5874 168 omc 0.4/ 0.2 0.1 0.0/ 0.0 22.5mb 1
zrnevhmx 9134 5874 154 omc 0.0/ 0.0 0.0 0.0/ 0.0 21.8mb 1
batmanmx 7652 5874 168 omc 0.0/ 0.0 0.0 0.0/ 0.0 20.7mb 4
midaemon 15146 1 -16 root 1.1/ 1.2 0.4 0.0/ 0.0 7.6mb 3
glance 15141 11414 154 root 1.7/ 2.6 0.9 0.0/ 0.0 7.0mb 1
vxfsd 34 0 138 root 0.2/ 0.2 0.1 33.8/36.4 1.8mb 58
c5xstdmx 15233 747 154 omc 1.9/ 1.9 0.1 1.1/ 1.1 420kb 1
ouiswsmx 15234 15233 154 omc 1.1/ 1.1 0.1 0.0/ 0.0 420kb 1
ps 15236 15235 149 root 1.1/ 1.1 0.1 0.3/ 0.3 296kb 1

MEMORY REPORT Users= 14
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 516 1945 99.2 136.9 220.0
Page In 154 611 29.6 43.0 85.0
Page Out 0 3 0.0 0.2 0.7
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 0kb 12kb 0.0 0.8 2.8
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 0 6 0.0 0.4 1.4

Total VM : 1.51gb Sys Mem : 218.4mb User Mem: 2.89gb Phys Mem: 4.00gb
Active VM: 695.3mb Buf Cache: 327.7mb Free Mem: 588.8mb

[RAC_1]

Load average in itself does not give exact picture of what is happening. The real picture would be looking at global priority queue. (You can check that with glance, gpm ,PV. )
following glance adviser syntax will also give it.
Prepare file as follows. (say glance.proc)
print "global priority queue", gbl_pri_queue

Now run following.

glance -adviser_only -j2 -iterations 3 -syntax glance.proc

If you see global priority queue high, then it may be aproblem, else do not worry as load average of 4 or 5 or say 6 with system havinf three cpus is fine.

The fast context switching jobs may also create high run queue and priority run queue.

sar -w 1 4
sar -c 1 4
sar -q 1 4

[zs_hzh]

thanks for responding so quickly ,RAC .

here is the result ,can you help me to check it ,
one more thing ,what caused user compaint that server is slow? i guess it should be long response time. then how to check it?

thanks

# sar -q 1 4

HP-UX nbnkscs1 B.11.00 A 9000/800 11/22/05

18:03:46 runq-sz %runocc swpq-sz %swpocc
18:03:47 2.0 50 0.0 0
18:03:48 1.0 50 0.0 0
18:03:49 1.0 50 0.0 0
18:03:50 1.0 51 0.0 0

Average 1.2 50 0.0 0
# sar -c 1 4

HP-UX nbnkscs1 B.11.00 A 9000/800 11/22/05

18:03:58 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
18:03:59 5108 184 17 2.97 2.97 648871 0
18:04:00 8872 318 26 12.12 11.11 1721148 42408
18:04:01 4452 170 22 0.00 0.00 7097 1014
18:04:02 3416 196 9 1.00 1.00 533504 410624

Average 5450 217 18 3.99 3.74 722693 113125
# sar -w 1 4

HP-UX nbnkscs1 B.11.00 A 9000/800 11/22/05

18:04:08 swpin/s bswin/s swpot/s bswot/s pswch/s
18:04:09 0.00 0.0 0.00 0.0 994
18:04:10 0.00 0.0 0.00 0.0 1300
18:04:11 0.00 0.0 0.00 0.0 1355
18:04:12 0.00 0.0 0.00 0.0 1072

Average 0.00 0.0 0.00 0.0 1179


nbnkscs1:/ (102) root% glance -adviser_only -j2 -iterations 3 -syntax glance.proc

Welcome to GlancePlus


global priority queue 0.0
global priority queue 0.0
global priority queue 0.0

[RAC_1]

Just do not worry. Your cpus are not doing big context switching. All is fine.

Other than this do you see any bottleneck?? cpu, disk, memory and network?? Again look at glance.

[zs_hzh]

thanks RAC ,
Glance don't say any bottleneck so far,
but our customer said the system is slowly.

i've just taken a look at DB ,seems DB server is having strongly DISK problems now.

maybe that's the reason of complaint.

but i'm still wondering if you could help me explain why load is so high when comparing to cpu usage?

what shoud i do first ,if i wanna tuning this server ? adding memory?

tks anyway!