Unbalanced processor load

[Paul Haygarth]

I have an rp7420 with 2 processors. The system was implemented last May, and ran fine, until the last couple of months. Since the last time we rebooted (91 days), the load on processor 1 is anywhere from 9.00 to 20.00, and the load on processor 2 is never above 1.00.

I've attached a screen shot of what it looks like. One thing I've noticed is that statdaemon is generally the top process.

Any ideas?

[Cesare Salvioni]

hi,
not an answer, that's really strange because cpu load (user and sys) is almost the same for both processor and, strange this too, is very low (idle about 80%)

So the question: is where does it come that high load average????
1. Sure it was not the same from the very beginning and you noticed id only right now?
2. Did you check for some patch solving this that, in my opionin looks like a top fake?
3. Try to observe the cpu with some other tool (like sar, glance or perfview) to confirm the top output

hope it helps

[Bill Hassell · Accepted solution]

Load average is not what you think it is. It is the average number of processes ready to run (the kernel's runqueue) during the measurement period. For long run-time processes (things humans can relate to), this would mean that a bunch of processes were ready to run but there was no free processor available. But that's not the case here as the second processor is not busy, and the first processor isn't busy either. This is a classic symptom of a bunch of short run-time processes that quickly start, check on something and go idle, perhaps in a few milliseconds.

I have seen workloads of 40 on a 2-processor system which was caused by some 250 copies of a polling program. And users were not complaining because the programs consumed very little CPU time. The key is that system overhead is high relative to user CPU time (50:50) which also points to programs that ask the kernel for something and then go idle. Now none of this is a bad thing unless it is unexpected. Look for lots of copies of the same program (like phantom?) and see what they are supposed to do.

[Tim D Fulford]

Hi

What you are seeing could be due to other bottlenecks. The runq is the number of running or runable processes (as the top line said 52 processes running!!!). As the cpu lod is low 17% or so, you may have a network of more likely disks bottleneck. This will mean alot of processes will be in the runable state.

I do not know why cpu0 is highly loaded and cpu1 is not, but are you using an application that uses processor affinities? If so this could be the culplrit. My pessimistic side would also to check out the health of CPU0 using "mstm" and double check it is not faulty..

Tim

[Sridhar Bhaskarla]

Hi Paul,

Well. I don't see your CPUs terribly unbalanced. CPU 0 was 87% and CPU 1 was 80% idle. You may not get 100% balancing always particularly if you have single threaded processes. If you pay close attention, on the average, 16 processes consumed only 13% CPU on proc#0 whereas .6 processes consumed 20% CPU on proc#1 so for me system did a good job to spread them that way.

I see atleast one process on CPU#0 zombie. There is a total of two zombies.

-Sri

[Steven E. Protter]

I don't think you really have a problem. Load factor is not a good way to measure processor issues. The idle percentages Sridar pointed out are a better way.

If you have idle time you don't have very busy processes.

Based on your original post I'd say your system really isn't working very hard at all.

You seem to have a lot of processes ready to run that don't need to run/

Might be intesting to see the system booted or some long term performance measurement(script attached).

SEP

[Tim D Fulford]

As I said previously High runQ / CPU load & low cpu utilisation is symtomatic of IO bottleneck.

What are the disks & or network doing.

Tim

[Paul Haygarth]

Thanks for all the replies everyone. Bill Hassell hit right on the nose.

After a little more digging to provide more information, I found a process that was spawning a child every second, to check a queue. I talked to the progammer who was doing this, and had him change the software to check once every 15 seconds, and the load on CPU0 dropped to less than 1.00