High Percent System Utilization

[Eric Singer]

We have a DL585 with 4 processors and 20GB of RAM and it becomes very busy every morning (Total CPU Utilization in the 80% to 100% range). When I bring up "top" it frequently shows a higher percentage under "system" utilization column than "user". Using top I can identify the highest CPU "User" process (which is normally in the 15% range), but is there anyway to find where processor usage is going on the "system" side.

[Steven E. Protter]

Shalom Eric,

A system can run 100% all the time and have no adverse effects.

There is no reason to worry or investigate unless the response of the system to the users is problematic.

You can collect more data with sar utils or even install HP's Glance GPM and look at many metrics if there is actually an issue.

SEP

[Eric Singer]

Hi Steven, thanks for the response.

Let me rephrase the question a little bit. Is there a way to tell the source of the percent system utilization?

Thanks,

Eric

[Ivan Ferreira · Accepted solution]

The source of the problem may be hard to find, because the system time normally wont appear in a "ps" because is time used to complete operating system functions.

Normally, the process withg highest CPU usage is generating more system activity. This could be I/O or paging. Check with vmstat your pagin statistics with vmstat and io statistics with iostat (you can use sar also).

You may use the /usr/bin/time command when you start a batch process. This will report the %time spend on user and system.

[Vitaly Karasik_1]

Check your morning's cronjobs.

See IBM RedBook for more info about Linux tuning http://www.redbooks.ibm.com/abstracts/redp3861.html?Open

Rgds,
Vitaly

[Eric Singer]

It does look like a cron that kicks off at around 2:00 AM. You can see how the system utilization starts going up while the user goes up some, but not nearly as much.

I appreciate all the help so far and I'll read that IBM Redpaper.

Here's a bit of the sar from this morning.

Time CPU %user %nice %system %iowait %idle
01:54:02 all 6.30 0.00 6.67 0.25 86.78
01:56:02 all 6.92 0.00 7.31 0.29 85.48
01:58:02 all 6.49 0.00 7.03 0.25 86.23
02:00:02 all 6.39 0.00 7.52 0.34 85.75
02:02:02 all 18.97 0.00 29.23 0.88 50.92
02:04:02 all 16.12 0.00 25.17 0.77 57.93
02:06:02 all 18.21 0.00 26.39 0.81 54.58
02:08:02 all 14.37 0.00 24.97 1.04 59.62
02:10:02 all 16.19 0.00 23.81 0.88 59.12
02:12:02 all 23.68 0.00 26.46 1.10 48.76
02:14:02 all 15.87 0.00 23.76 0.85 59.52
02:16:02 all 18.61 0.00 24.11 0.89 56.39
02:18:02 all 15.32 0.00 22.80 0.83 61.06
02:20:02 all 16.32 0.00 24.25 0.90 58.53
02:22:02 all 16.00 0.00 24.47 0.84 58.68
02:24:02 all 12.96 0.00 22.65 0.95 63.44
02:26:02 all 13.55 0.00 23.95 0.85 61.65
02:28:02 all 13.42 0.00 22.59 0.91 63.07
02:30:02 all 13.73 0.00 26.36 0.83 59.08
02:32:02 all 18.93 0.00 25.56 0.82 54.70
02:34:02 all 13.82 0.00 24.33 0.87 60.98
02:36:02 all 20.37 0.00 28.20 1.20 50.24
02:38:02 all 14.55 0.00 31.26 1.80 52.39
02:40:02 all 14.26 0.00 30.19 2.16 53.38
02:42:02 all 15.03 0.00 25.77 0.90 58.30
02:44:02 all 15.36 0.00 25.75 0.85 58.03
02:46:02 all 22.35 0.00 27.84 0.90 48.91
02:48:02 all 15.79 0.00 25.36 0.86 58.00
02:50:02 all 17.31 0.00 26.75 0.84 55.10
02:52:02 all 16.96 0.00 24.80 0.83 57.42
02:54:02 all 14.69 0.00 23.99 0.91 60.41
02:56:02 all 14.41 0.00 24.87 0.81 59.92
02:58:02 all 13.64 0.00 22.23 0.91 63.23
03:00:02 all 16.32 0.00 24.94 1.79 56.96
03:02:02 all 22.90 0.00 41.25 1.14 34.71
03:04:02 all 23.14 0.00 41.89 1.57 33.39
03:06:02 all 35.49 0.00 37.48 1.65 25.37
03:08:02 all 28.93 0.00 27.00 2.69 41.37
03:10:02 all 26.21 0.00 28.63 2.39 42.76
03:12:02 all 32.64 0.00 24.63 2.58 40.16
03:14:02 all 20.07 0.00 24.34 2.55 53.04
03:16:02 all 32.79 0.00 22.71 2.27 42.22
03:18:02 all 29.43 0.00 20.98 2.45 47.13
03:20:02 all 27.02 0.00 22.83 2.26 47.90
03:22:02 all 35.12 0.00 22.56 1.41 40.91
03:24:02 all 20.31 0.00 21.16 1.75 56.78
03:26:02 all 19.80 0.00 21.85 1.73 56.63
03:28:02 all 18.99 0.00 20.31 1.81 58.89
03:30:02 all 26.10 0.00 23.69 1.79 48.42
03:32:02 all 24.07 0.00 36.06 1.48 38.39
03:34:02 all 16.35 0.00 36.10 1.41 46.15
03:36:02 all 22.74 0.00 38.62 1.11 37.52
03:38:02 all 16.91 0.00 37.02 1.34 44.73
03:40:02 all 17.07 0.00 40.00 1.19 41.74
03:42:02 all 17.16 0.00 40.77 1.19 40.88
03:44:02 all 15.98 0.00 36.88 1.31 45.82
03:46:02 all 21.10 0.00 38.29 1.17 39.44
03:48:02 all 16.75 0.00 38.20 1.27 43.79
03:50:02 all 19.78 0.00 39.36 1.16 39.69
03:52:02 all 17.89 0.00 38.24 1.14 42.73
03:54:02 all 16.87 0.00 38.75 1.33 43.06
03:56:02 all 16.81 0.00 38.32 1.32 43.54
03:58:02 all 16.67 0.00 38.71 1.24 43.39
04:00:02 all 16.37 0.00 38.39 1.11 44.13

[Alan_152]

Run "top", and "renice" some of the process hogs.

[Mike Stroyan]

The "user" and "system" time for each process is present in /proc//stat as documented in "man proc". You could sort by those fields to see which running process has used the most "system" time.

echo " sys user pid cmd";awk '{printf "%7d %7d %7d %s\n",$15,$14,$1,$2}' /proc/[0-9]*/stat | sort -bn | tail -20

You may find a lot of system time in kernel processes like kjournald or reiserfsd. The root cause of such high system utilization would be in some user process. In the case of the file system processes they would be so busy because some user process did a lot of file system writes.

[Eric Singer]

Hi Mike,

Thanks for the command, I can definitely use this to get a process breakdown.

Eric