High CPU usage on N-class

[Mihails Nikitins]

Hi,

I have the following problem. Summary of 'top' utility always shows extremely high usage of cpu1 on N4000 with 4 processors (HP-UX 11.0). Its SYS column is always greater than 60%.

System: sys1 Thu Dec 13 16:42:02 2001
Load averages: 0.49, 0.52, 0.54
96 processes: 93 sleeping, 3 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.85 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 1.00 19.4% 0.0% 68.7% 12.0% 0.0% 0.0% 0.0% 0.0%
2 0.10 0.4% 0.0% 3.0% 96.6% 0.0% 0.0% 0.0% 0.0%
3 0.00 0.4% 0.0% 1.6% 98.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.49 5.0% 0.0% 18.4% 76.6% 0.0% 0.0% 0.0% 0.0%

However, none of processes listed by 'top' consumes much of cpu1. I have another another identical system that have the same number of processes. So, there are no thousands of small evil processes eating cpu1. CPU load of the second system is quite low.

Has anyone experienced abnormal cpu load on N-class? What tools can you advise to find out _what_ is happening on cpu1?

Thanks in advance for your comments!

BR,
Mihails Nikitin

[fg_1 · Accepted solution]

Mihails

If you have glance or glance plus running, you can drill down into cpu reports by processor and discover what is running and taking up the most load on that cpu.

Also, Sar -c will give you cpu reports.

The n-class usually does not have this situation occurr because of hw, it's usually a process that is a runaway, or a database that has an open query repeating.

Hope this helps.

Good luck.

[Helen French]

Hi,

You can closely watch all processes, loads and CPU usage through Glance.

See man pages of glance and gpm

HTH,
Shiju

[Sridhar Bhaskarla]

HI Mihails,

High system usage is due to a lot other reasons.

1. Heavy I/O activity causing disk bottlenecks
Do a sar and look at %wio. It shoudln't be more than 15%. Also look at sar -d 2 20 and observe the %busy, it shouldn't be constantly at above 80% for any of the disks. avque should be 0.5
2. High memory usage. Do a swapinfo -t and look at the swap used column of each swap volume. Not the reserved row.
3. The application itself may be doing a lot of system calls like open, close etc.,

-Sri

[Sridhar Bhaskarla]

Forgot one more thing.

Make sure you have latest diagnostic patches and in general all the patches. So, some of the daemons like cclogd, mib2agt etc seemed to create a lot of CPU activity.

-Sri

[Paula J Frazer-Campbell]

Hi

Looking at your TOP output your system is quiet - 96 processes 3 running .
The the 68.7% busy just reflects that that cpu is being used by probably a system file ??? ie syncer, pwgrd etc.

On a quiet system this is not unusual to see this happen as user requirements go low then an apparent normally low usage programme will be given more room to breath.

If you have glance installed it is by far a better tool than top to monitor what is going on on your server.

-----------------------------

Here is an output from a N class with 725 users logged in via ip and x25.

It looks busy and it is, it is also close to time to give it more memory and cpu power.




System: N-0 Mon Jan 14 17:40:33 2002
Load averages: 4.54, 3.58, 2.75
1777 processes: 1763 sleeping, 14 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 5.64 14.4% 0.8% 7.2% 77.6% 0.0% 0.0% 0.0% 0.0%
3 3.96 14.4% 1.8% 10.1% 73.7% 0.0% 0.0% 0.0% 0.0%
4 4.40 18.1% 0.0% 8.4% 73.5% 0.0% 0.0% 0.0% 0.0%
5 4.18 17.9% 0.2% 5.6% 76.3% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 4.54 16.0% 0.8% 8.0% 75.3% 0.0% 0.0% 0.0% 0.0%


Paula

[Rusty Sapper]

Hi,
I have similar situations on the N-classes that I have. In my situation, it is because some of the processes are multi-threaded processes. I don't think top takes into consideration multiple threads.

ie.
if you have 4 processes, each running on a different processor, but the one on CPU4 has 32 threads, then that CPU is going to show a much greater load.

Glance should be able to tell you how many threads a process is using.



-Rusty

[Mihails Nikitins]

Thanks for you ideas, I'll try to apply them!

[karthikeyan_2]

Hi,
I am also facing the same problem.When i tried to see the multithreded process's CPU usage using top command , it show more than 100 % CPU usage.But in glancePlus(gpm),it shows less than 10 % CPU usage in a muli processor system.

Can anyone please tell me why top command and glance are showing diffrent data for CPU usage ?

Thanks
Karthik

[Bill Hassell]

top is a very old utility that uses classic kernel values to try to guess what the CPU is actually doing. The problem is that top simply can't track all the tasks that are going on as it is just another process and tries to simplify the kernel's tasks into crude buckets (processes, system, I/O). And it tries to do this while formatting the data and writing to the screen. While this seems simple, several processes can context-switch during just 1 line of text on the screen.

The reason that Glance is so much more accurate is midaemon and kernel hooks that provide modern kernel metrics. midaemon grabs the information very quickly and places the data into shared memory for Glance to process. Glance takes special care to account for threads and the complexities of multi-processor systems.

As far as system overhead, 60%, even 80% may be normal is the tasks being performed are numerous (dozens, hundreds) and perform very short I/O, perhaps a LAN polling task. The I/O is short and takes very little time to complete and when thousands of little I/Os are performed by hundreds of programs, system overhead is very high...and normal.

[Mark Hudson_1]

I just wanted to submit a suggestion to download patch PHSS_23659 (for HP/UX 11.11) which actually addresses a bug in the cclogd (and other components)
This is also discussed in this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x7b707bb04b5cd611abdb0090277a778c,00.html

Cheers.