topic Re: High system cpu in Operating System - HP-UX

High system cpu

Eric Guerizec — Tue, 06 Aug 2002 14:52:06 GMT

Hello,
I have high system cpu on a HP N4000 4 processors.

Look at this, sar 2 10 :
HP-UX myserver B.11.00 U 9000/800 08/06/02

17:32:55 %usr %sys %wio %idle
17:32:57 41 26 5 28
17:32:59 41 35 1 22
17:33:01 42 35 3 19
17:33:03 44 34 4 19
17:33:05 60 26 3 12
17:33:07 58 30 1 11
17:33:09 59 27 1 12
17:33:11 50 28 2 20
17:33:13 20 41 4 34
17:33:15 18 39 4 39

Average 43 32 3 22

vmstat 2 10 :

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
4 0 0 28589 824109 6 4 0 0 4 0 0 0 113 5 12 8 80
4 0 0 28589 824059 3089 2001 3 0 2018 0 0 1760 56331 2986 21 41 38
4 0 0 28589 824268 3172 2059 1 0 2073 0 0 1705 55201 2816 22 38 40
9 0 0 27256 824161 2975 2007 0 0 2012 0 0 1869 51669 2742 22 33 46
9 0 0 27256 823810 2913 2023 0 0 1996 0 0 1829 50947 2577 19 31 50
1 1 0 27874 823940 2980 1961 0 0 1962 0 0 1715 50822 2527 20 42 39
1 1 0 27874 824220 2936 1993 0 0 2008 0 0 1663 50113 2443 22 35 43
6 0 0 29439 824147 2945 1929 0 0 1940 0 0 1615 50384 2408 16 37 47
6 0 0 29439 823905 2836 1797 0 0 1768 0 0 1728 48562 2576 29 27 44
6 0 0 29439 824130 2802 1918 0 0 1926 0 0 1724 48683 2612 27 43 29

I think there is too many Page reclaims!

Now, what do I have to do with glance to find the process which consumes the system cpu?

With glance, I suspect a user script. Whan I look at the Process Ressources for this shell script, I find values like this :
Virtual Faults = 9684
Forks & VForks = 1133
Process Wait States shows 24% on PIPE.

Any ideas, comments ???
Thanks!

Eric

Re: High system cpu

Bill McNAMARA_1 — Tue, 06 Aug 2002 14:57:19 GMT

use gpm (GUI for glance)
Then double click on any process name.
via the menu option reports, you can drill into the process.. you can do so also in text base via the soft keys.

PS: when including formatted text in the forums do so as an attachment... it's nearly impossible to read otherwise...

Later,
Bill

Re: High system cpu

Hai Nguyen_1 — Tue, 06 Aug 2002 15:01:43 GMT

Eric,

I think your system may be low on RAM based on the sar and vmstat outputs since the sys% is pretty high in sar (average 35%.)

Hai

Re: High system cpu

Stefan Farrelly — Tue, 06 Aug 2002 15:10:30 GMT

You have tons of free memory from your vmstat output and pi and po (page in and pageout) are zero so you have NO memory problems/pressures.

For CPU run top, this will show you who is using it all.

Re: High system cpu

Steven Gillard_2 — Tue, 06 Aug 2002 15:42:06 GMT

Certainly doesn't look like a memory issue. Another common cause of high system cpu usage is directories with huge numbers of files. Running "sar -a" will tell you how many directory blocks per second are being scanned - if this number is consistently >1000 that may be the problem.

You can use the find command to locate large directories:

# find / -type d -size +128

This will report directories larger than 64k in size.

Otherwise as others have suggested try to find out which processes are causing the high CPU utilisation, then use glance or tusc to find out what those processes are doing lots of.

Regards,
Steve

Re: High system cpu

Eric Guerizec — Tue, 06 Aug 2002 17:34:01 GMT

Steven; I think you are on the right way!
sar -a says :
iget/s = 25
namei/s > 3500
dirbk/s > 1000
Now, I am sure that an application script (launched by cron) is responsible for the problem! If I stop this crontab then sar -a says :
iget/s = 2
namei/s > 200
dirbk/s > 100

Do you known what must change the user in his script to avoid my problem (high sys cpu)?
Find command brought back some repertories but which don't belong to the user homedir!
Directories are :
/var/rbootd
/var/spool/mqueue
and some others user subdir.

Re: High system cpu

Bill Hassell — Tue, 06 Aug 2002 22:54:30 GMT

The high inode activity is caused by the script--which you've already proved. You can duplicate (maybe even make it worse) by doing:

du -s /

This will make the opsystem and the disk VERY busy with directory and inode gets very high. That's perfeectly normal since the du command is summarizing all the directories. Now if your user decided to run bdf and du over and over again, then certainly you'll see this behavior. It's caused by performing directory searches.

Re: High system cpu

Steven Gillard_2 — Wed, 07 Aug 2002 08:30:01 GMT

You'll need to have a closer look at the script, and probably rewrite it so its more efficient.

Complex and badly written shell scripts can certainly have a large impact on system performance as they have a habit of spawning a large number of processes. The best way to fix this is to rewrite your complex shell scripts in perl!

Regards,
Steve

Re: High system cpu

Eric Guerizec — Wed, 07 Aug 2002 15:37:08 GMT

The user execute his script every 1 minute or every 5 minutes (crontab). The script makes many calls to system commands like tail, grep, cat... But /usr/bin isn't in first position in the $PATH. I think that it could be the cause of the problem (many namei/s and dirbk/s)
What do you think about it?