CPU Load seems to be "stuck" at 1

[Brian Bientz]

Here's a snapshot from a top command:
System:

devdb1 Wed Feb 18 08:36:51 2004
Load averages: 0.59, 0.58, 0.58
165 processes: 161 sleeping, 4 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.08 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
1 1.10 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.59 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%

Memory: 85536K (39272K) real, 274516K (144564K) virtual, 20244K free Page# 1/15


One thing I have noticed is that one CPU (on all three of our systems) is consistantly "stuck" with a load of 1+. However, as you can see on this dev machine, the CPU is nearly 100% idle.

After shutting down all the database processes the load value returns to near 0. Has anyone seen this before? Why would the load value always report 1+ if the CPU is nearly idle?

[Pete Randall]

Brian,

You don't mention what your DB is, but I see similar results on my Informix development machine. It's relatively idle, yet the load on one CPU of the two is consistently above one:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.04 53.9% 0.0% 3.4% 42.7% 0.0% 0.0% 0.0% 0.0%
1 0.54 51.5% 0.0% 1.4% 47.1% 0.0% 0.0% 0.0% 0.0%

I would assume that this is normal DB housekeeping activity - I know my programmers aren't doing that much!


Pete

[Bill Hassell]

If you use top or Glance, you can track down which process(es) are using time. This is a programming issue, not something you can change in HP-UX. Some programmers have been known (right after Unix for Dummies training) to issue select() calls and rather than wait, go into an infinite loop polling the open calls for results. Recoding the app changes the load from 100% CPU (user+system) to 0%. Perhaps there is a config parameter that makes the programs behave more reasonably. On the other hand, computers are always running full speed. They are either executing the idle loop waiting for something to happen or they are running user programs or some combination of the two.

[Steven E. Protter]

You might want to search patches for top patches or:

Try glance or sar scripts and see what they show.

Attaching an example.

SEP

[Jeff Schussele]

Hi Brian,

And you bought these systems to do what?....

I ALWAYS say if the users aren't complaing.....

IF one CPU is pegged tell the coders to think about multi-threading the app.

My $0.02,
Jeff

[Brian Bientz]

If you notice, the CPU is not pegged. To the contrary ... the CPU is nearly idle according to the IDLE column in the top command.

However, the LOAD column consistantly shows a value > 1.

What does this mean? How can the CPU be 100% idle but always have a load > 1.

On our production system we see a similar issue. Granted, in that case the CPU's are not nearly 100% idle but the LOAD column always seems to be inflated by 1.

Just wierd ...

[Bill Hassell]

CPU load average does not mean CPU load. It means average runqueue depth. And the metric is only sampled. The runqueue is the queue of programs that are running or ready to run. If 10 programs that do no I/O are all running at the same time, the runqueue depth will be 10 even though there may be only 2 processors in the box. Through context switching, each program will get a short period of processing time, but each processor is running at 100% (which is the max).

What about 0% user, 99% idle and runqueue is greater than 1? Most likely there is a process that is polling (perhaps a LAN task such as a socket) fairly rapidly, say once every 100ms. The process consumes virtually no CPU time, the opsystem doesn't need very much time to send out the packet, so most of the time is spent waiting on I/O completion--yet the runqueue metric almost always shows a load of one program since it wakes up so many times each second.

The load average (uptime or top) is not a very meaningful metric until compared with other metrics (avg user, nice, sys times).

[Brian Bientz]

Thanks Bill ... great information. Now you know what I'm going to ask next! Is there way to identify these processes?