top and sar shows 100% with 80% as SYS

[Dinesh R]

Hi all,
we use hp-ux B.11.11.
top and sar shows 100% CPU with 80% as SYS
but none of the sys process are shown in the top output
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
1.61 18.8% 5.0% 76.2% 0.0% 0.0% 0.0% 0.0% 0.0%

Is this a bug or any other workaround available for this

[Safarali]

Hi Dinesh

You have to send sar -b 5 5
sar -u 5 5
and top output

Regards
Safar

[Dinesh R]

Please find the output
sar -b 5 5

HP-UX xxxxxx B.11.11 U 9000/800 02/10/10

14:39:27 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
14:39:32 6 5779 100 16 699 98 0 0
14:39:37 4 1727 100 6 28 78 0 0
14:39:42 6 3616 100 5 40 88 0 0
14:39:47 8 1150 99 2 7 68 0 0
14:39:52 5 3210 100 4 71 94 0 0

Average 6 3097 100 7 169 96 0 0

sar -u 5 5

HP-UX xxxxx B.11.11 U 9000/800 02/10/10

14:41:16 %usr %sys %wio %idle
14:41:21 24 76 0 0
14:41:26 24 76 0 0
14:41:31 22 78 0 0
14:41:36 28 72 0 0
14:41:41 28 72 0 0

Average 25 75 0 0


System: xxxxxx Wed Feb 10 14:42:07 2010
Load averages: 1.76, 1.72, 1.66
1479 processes: 1410 sleeping, 68 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 2.96 58.4% 12.9% 28.7% 0.0% 0.0% 0.0% 0.0% 0.0%
1 2.33 28.7% 7.9% 63.4% 0.0% 0.0% 0.0% 0.0% 0.0%
2 2.19 53.5% 0.0% 46.5% 0.0% 0.0% 0.0% 0.0% 0.0%
3 1.86 5.0% 0.0% 95.0% 0.0% 0.0% 0.0% 0.0% 0.0%
4 2.26 11.9% 0.0% 88.1% 0.0% 0.0% 0.0% 0.0% 0.0%
5 2.41 6.9% 0.0% 93.1% 0.0% 0.0% 0.0% 0.0% 0.0%
6 1.80 6.9% 0.0% 93.1% 0.0% 0.0% 0.0% 0.0% 0.0%
7 3.17 11.9% 0.0% 88.1% 0.0% 0.0% 0.0% 0.0% 0.0%
8 2.63 21.8% 0.0% 78.2% 0.0% 0.0% 0.0% 0.0% 0.0%
9 1.90 20.8% 1.0% 78.2% 0.0% 0.0% 0.0% 0.0% 0.0%
10 1.47 81.2% 3.0% 15.8% 0.0% 0.0% 0.0% 0.0% 0.0%
11 1.47 25.7% 2.0% 72.3% 0.0% 0.0% 0.0% 0.0% 0.0%
12 1.68 4.0% 0.0% 96.0% 0.0% 0.0% 0.0% 0.0% 0.0%
13 1.63 13.9% 0.0% 86.1% 0.0% 0.0% 0.0% 0.0% 0.0%
14 1.36 26.7% 0.0% 73.3% 0.0% 0.0% 0.0% 0.0% 0.0%
15 1.74 5.0% 1.0% 94.1% 0.0% 0.0% 0.0% 0.0% 0.0%
16 1.08 10.9% 1.0% 88.1% 0.0% 0.0% 0.0% 0.0% 0.0%
17 1.17 5.9% 1.0% 93.1% 0.0% 0.0% 0.0% 0.0% 0.0%
18 1.15 5.9% 0.0% 94.1% 0.0% 0.0% 0.0% 0.0% 0.0%
19 1.39 25.7% 1.0% 73.3% 0.0% 0.0% 0.0% 0.0% 0.0%
20 1.06 7.9% 0.0% 92.1% 0.0% 0.0% 0.0% 0.0% 0.0%
21 1.22 10.9% 0.0% 89.1% 0.0% 0.0% 0.0% 0.0% 0.0%
22 1.04 12.9% 0.0% 87.1% 0.0% 0.0% 0.0% 0.0% 0.0%
23 1.24 9.9% 0.0% 90.1% 0.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.76 19.8% 1.0% 79.2% 0.0% 0.0% 0.0% 0.0% 0.0%

Memory: 26846088K (15973608K) real, 32349960K (19390316K) virtual, 8910976K free Page# 1/60

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
22 ? 23010 oracle 241 20 5298M 5568K run 17387:57 101.08 100.90 oracleinstance1
20 ? 12547 oracle 241 20 2204M 7360K run 265:19 100.82 100.65 oracleinstance2
17 ? 12505 oracle 241 20 5299M 6080K run 129907:46 100.15 99.98 oracleinstance1
18 ? 2243 oracle 241 20 2203M 6336K run 269:19 100.10 99.92 oracleinstance2
23 ? 10155 oracle 241 20 5299M 6272K run 266:50 99.35 99.18 oracleinstance1
16 ? 23072 oracle 241 20 5298M 5568K run 17396:20 97.64 97.47 oracleinstance1
1 ? 23068 oracle 241 20 5298M 5568K run 17387:29 97.10 96.93 oracleinstance1
15 ? 15947 oracle 241 20 5298M 5632K run 260:33 92.96 92.80 oracleinstance1
3 ? 2163 oracle 241 20 2202M 6208K run 274:10 91.72 91.56 oracleinstance2
5 ? 17484 oracle 241 20 2202M 6208K run 260:00 91.08 90.93 oracleinstance2
13 ? 13586 oracle 241 20 2202M 6208K run 129932:35 89.40 89.24 oracleinstance2
6 ? 22031 oracle 241 20 2201M 5184K run 272:46 87.93 87.78 oracleinstance3
7 ? 21996 oracle 241 20 2202M 6208K run 271:20 87.51 87.36 oracleinstance3
12 ? 2235 oracle 241 20 2202M 6208K run 266:21 86.85 86.70 oracleinstance2
8 ? 23019 oracle 241 20 5299M 5760K run 17402:18 85.85 85.70 oracleinstance1
19 ? 23014 oracle 241 20 5298M 5568K run 17394:38 85.50 85.35 oracleinstance1
11 ? 2226 oracle 241 20 2203M 6272K run 268:00 85.22 85.07 oracleinstance2
9 ? 18344 oracle 241 20 2201M 5184K run 270:37 82.12 81.98 oracleinstance3
14 ? 23095 oracle 241 20 5299M 6016K run 17391:44 81.79 81.65 oracleinstance1
4 ? 22017 oracle 241 20 2201M 5184K run 263:49 78.57 78.44 oracleinstance3
21 ? 22035 oracle 241 20 2203M 6848K run 270:25 76.97 76.84 oracleinstance3
0 ? 22026 oracle 241 20 2201M 5184K run 278:00 65.71 65.59 oracleinstance3
2 ? 29456 oracle 241 20 5304M 8240K run 2:00 39.16 39.10 oracleinstance1
10 ? 203 oracle 241 20 5303M 8560K run 1:54 38.55 38.48 oracleinstance1
1 ? 28975 oracle 241 20 2203M 6320K run 40:38 19.85 19.81 oracleinstance3

[Fabian Briseño]

Hello Dinesh.
Also post ouput of vmstat comand.

vmstat 3 10

Question, do you have something scheduled in crontab which might be executing constantly?

Is this activity constnat or does it happen at certain times?

Since when is this occurring?

[Dinesh R]

Hi Fab,
Please find the output of vmstat
vmstat 3 10
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
32 412 0 4879674 2340748 501 92 0 0 0 0 0 16612 412525 3930 13 5 82
32 412 0 4879674 2340288 78 33 0 0 0 0 0 13767 1642695 9393 22 77 1
32 412 0 4879674 2341152 95 39 0 0 0 0 0 12226 1578553 7938 21 79 0
32 415 0 4773287 2340085 218 68 0 0 0 0 0 12742 1612515 8266 19 80 1
32 415 0 4773287 2341638 233 65 0 0 0 0 0 11578 1558160 7128 19 80 1
32 415 0 4773287 2342054 439 72 0 0 0 0 0 12707 1604526 8212 19 80 1
36 407 0 4784451 2339321 270 36 0 0 0 0 0 11870 1564905 7490 20 80 0
36 407 0 4784451 2341462 139 17 0 0 0 0 0 12363 1605207 8189 20 79 0
36 407 0 4784451 2345430 157 51 0 0 0 0 0 12007 1614236 7807 20 79 1
36 407 0 4784451 2345983 269 55 0 0 0 0 0 12427 1586147 8291 20 79 1


This is occuring for past few days. The cpu is constant at 100% for the past 5 days.
We tried killing many oracle process but did not work.
The actual problem seem to be with the sys showing as 80%. Normally it is 6 to 7 %. The user process consumes only 20%. This is also normal. And no user has complained of any slowness in the system.
Can this be a bug in top and sar command..?

[Dinesh R]

Also nothing in crontab which is suspicious.
Only some scripts are there that run daily once

[Fabian Briseño]

Hello Dinesh.
At first glance it states there is a memory shortage.

vmstat
r 32 = runnable procs. (in run queue)
is it 4 or 412? b 4 = # of procs. blocked from executing because they are waiting on I/O or memory
is it 12 or 0? w 12 = Processes waiting because they were swapped out, this could indicate a memory shortage

re = page reaclaims, a large # here shows a memory shortage


post outputs of commands

sar -d 3 10

swapinfo -tam

Can this be a bug in top and sar command?
It could be, have you searched for the latest patches on top and sar. But in the other hand these are two commands and they are giving you the same feedback.


What do these script that execute daily do?

Have a great day.