Why is sys cpu so high?

[Tom Terrian]

Background: HPUX 11.11, 4 cpu's.

For the majority of the day yesterday the HP box ran fine. However, from 10am to 11am the sys cpu numbers from top when to > 90% and the user processes slowed to a crawl. At 11am sys cpu when down to around 13% and the user processes ran normally.

What could cause sys cpu go so high for so long? What does sys cpu accually mean (i.e. what was going on)? Can this be caused by users or is there an O/S problem that I need to address?

Next time it happens, I would like to address it quickly.

Thanks,
Tom

[Rajesh G. Ghone]

Hi Tom,
I believe SYS CPU means the resources occupied by the system process,Is there any cron jobs assigned between 10-11am??.If yes then I would like to know what job it is,well i dont know weather cron takes sys or usr, but i believe it must be taking sys resources.

Regards,
Rajesh G.

[Bruno Vidal]

Hi,
sys cpu is the time spend on "system call". Ie the time spend to open/read/write/memory managment/etc.... each function that UNIX is UNIX. So a so hight system time means probably that your system was doing something wrong: trashing because too few memory, trashing buffer cache, or lan driver trashing because too many packet in, etc.... The problems can be also a user problem: try to create a small C prog that do always fork/exit and your system will be completly down (fork is really time consuming for kernel). Because we don't have any cristal ball, I thnink the best is to have an eye on tools like top/glance/MWA in order to determine what was wrong at this time, in order to determine which syscall(s) was eating all CPU.

Cheers.

[Tom Terrian]

No, there did not appear to be anything special from 10 to 11. The users said that they were not running any more or unusual processes. But the box just died for about an hour then picked up again.

[Massimo Bianchi]

Hi,
what process had the highest cpu slice ?

May be with this information we could understand better your problem.

Massimo

[Paula J Frazer-Campbell]

Hi

Setup and use sar - man sar - also search the forum for sar.

Without data from when the machine ran slow we are only guessing as to what slowed it down.

Paula

[Jeff Schussele]

Hi Tom,

Unfortunately MUCH more info would be needed to even begin to guess what happened.
But, if CPU usage went WAY up and the vast majority was SYS, then something was going on that caused the kernel to monopolize the CPUs.
We'd need to know what memory, swap & disk I/O was at that time to give you any hints.
I'd suspect that something either broke or ranaway & caused the kernel to spend enormous amounts of time handling it.

Rgds,
Jeff

[Bill Hassell]

AS mentioned, the system CPU time is used when application programs ask the kernel to do something and it is certainly possible to force the kernel to do a bunch of useless things. For instance, an infinite loop in a shell script that just asks for the date and dumps the results into /dev/null. The problem is locating the offending program(s).

It is also possible that a program that performs network queries (perhaps Xwindows) goes awry and generates hundreds of queries per second. The high system overhead would also be accompanied by high LAN traffic. This often happens with Xwindows and unstable PC-based Xwindow emulators, or unstable users that trash their PC rather than gracefully signing off from all remote applications. A runaway Xwindow program can generate 40% system overhead and hundreds of packets/sec.

[Tom Terrian]

Thanks for all of your comments. We will investigate it futher and if we find something I will update this thread.

[John Meissner]

there are several things that cold effect the cpu speed like that.

-running backups
-applications (oracle/SAP/etc..)
-intensive scirpts
-even performance monitors can effect the systems performance

I would recommend using the tools perfview and measureware. This can let you know exactly what is gong on. you can look at the most minute details of the systems performance and what processes are running. We use this tool and it's fantastic.