Anomalous behaviour: high CPU load with High IDLE %

[Marco Scala]

Hi Everyone!
On our development web server, an L Serie server, we have this strange behaviour. The "top" shows an average CPU Load of 15 or more processes, with a IDLE % very close to 100%. This is server is mainly used as the web server for the development teams. Since a couple of months, one team has now an application running on that server and they are complaining for very poor response time.

Do you think this is a good picture: having high load with high idle CPU %, or is it the symptom of something bad?

Here follows a tipical 'top' output for the server.
Regards,

Marco Scala
============================
System: swwxxxx Fri Aug 17 12:16:53 2001
Load averages: 22.51, 27.56, 16.39
154 processes: 146 sleeping, 8 running
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
22.51 0.0% 0.0% 2.0% 98.0% 0.0% 0.0% 0.0% 0.0%

Memory: 178668K (44912K) real, 256196K (187628K) virtual, 4656K free Page# 1/10


TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
? 4920 e060672 152 24 7888K 6796K run 8:16 2.35 2.34 wsiserver
? 28 root 152 20 0K 0K run 2:04 1.22 1.21 vxfsd
? 2 root 128 20 0K 0K sleep 1:30 0.86 0.85 vhand
? 1989 www 154 20 12660K 2428K sleep 9:30 0.68 0.68 ns-httpd
? 1984 www 154 20 11140K 1992K sleep 9:14 0.64 0.61 ns-httpd
? 1954 www 154 20 10660K 1948K sleep 9:17 0.61 0.61 ns-httpd
? 1894 www 154 20 11140K 2112K sleep 9:15 0.60 0.60 ns-httpd
? 1931 www 154 20 10548K 1660K sleep 8:58 0.60 0.59 ns-httpd
? 1905 www 154 20 10548K 2100K sleep 8:59 0.60 0.59 ns-httpd
pts/2 14387 root 154 10 5104K 3368K sleep 0:03 0.60 0.59 gpm
? 1995 www 154 20 11044K 2288K sleep 9:25 0.60 0.59 ns-httpd
? 1976 www 154 20 12116K 1512K sleep 9:44 0.59 0.59 ns-httpd
? 1948 www 154 20 10548K 2036K sleep 8:53 0.58 0.58 ns-httpd
? 1971 www 154 20 12884K 2260K sleep 9:24 0.58 0.57 ns-httpd

[Thierry Poels_1]

hi,
at first sight you have little free memory left. Maybe the server is swapping like hell, although I would then expect a higher system cpu percentage.

"swapinfo" and check swap usage,
"sar -u" and check %wio,
"sar -d" and check for heavy disk load,
...

good luck,
Thierry.

[Praveen Bezawada]

HI
This kind of behaviour is quite possible if the process are in wait states. CPU load will be high if the process/threads are actually doing something,otherwise if the processes are waiting for some IO etc. the amount of CPU used will be quite less.
So it is important to check the disk utilisation,memory utilisation and the states of the processes.

...BPK...

[Tim D Fulford]

Marco

The suggestions above are what I would (effectively) do. But just to clarify what could be happening here:

You probably have a bottle neck (memory, disk, swap, network.....) this is removing load from your cpu's (as it is queued somewhere else)

I would also do a couple of tests etc
o Runing less netscapes if possible & look at the load then
o Is there a performance problem from the users point of view?
o If you have MeasureWare look at GBL_ALIVE_PROC
GBL_ACTIVE_PROC
GBL_STARTED_PROC
GBL_COMPLETED_PROC
GBL_PRI_QUEUE
GBL_RUN_QUEUE
GBL_MEM_QUEUE
GBL_IPC_SUBSYSTEM_QUEUE
GBL_NETWORK_SUBSYSTEM_QUEUE
GBL_SLEEP_QUEUE
GBL_DISK_SUBSYSTEM_QUEUE
GBL_NETWORK_SUBSYSTEM_QUEUE
Alternatively (if you have it) run glance

Good luck

Tim

[Sridhar Bhaskarla]

uptime and cpu load represent run queue not exactly the queue. You may want to check with
sar -q and see if there are any processes that are getting swapped out with %swpocc. If it is zero, then there is no problem.

I feel there are a lot of very light processes but are getting processed within no time so we are not seeing any utilization. They may be waiting on something (mostly IPC).

Keep an eye on GBL_PRI_QUEUE (Actual processes in queue that is waiting to be run) and IPC_SUBSYSTEM_QUEUE. If there is a contention within the IPC also this could happen.

-Sri