topic Re: %CPU 100% with %SYS > 70% in Operating System - HP-UX

%CPU 100% with %SYS > 70%

Pedro Cirne — Tue, 14 Jun 2005 11:48:13 GMT

Hi,

I'm having a performance problem with an L2000-44. The CPU is allways at 100%, but the strange thing is that %SYS is allways greater than 70%. Any ideia of what the problem might be? Some statistics attatched :-)

Thank you!

Re: %CPU 100% with %SYS > 70%

Pete Randall — Tue, 14 Jun 2005 11:55:29 GMT

Are you swapping? Run "swapinfo -tam" and "vmstat 1 5" and post the output.

Pete

Re: %CPU 100% with %SYS > 70%

RAC_1 — Tue, 14 Jun 2005 11:57:17 GMT

What are those mpbe processes and multiux processes??

Can you post following??

UNIX95= ps -ef -o "pid, ppid, args, cpu" | sort -nk4 | head -15

Anil

Re: %CPU 100% with %SYS > 70%

A. Clay Stephenson — Tue, 14 Jun 2005 12:02:43 GMT

You have a rather high value for process switches/second. I suspect that you have a number of small processes that are being spawned and executed. THe other thing that could cause this is a very low timeslice setting. What is your timeslice value?

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Tue, 14 Jun 2005 12:05:52 GMT

Hi,

Attahced find the statistics requested.

Thks

Re: %CPU 100% with %SYS > 70%

RAC_1 — Tue, 14 Jun 2005 12:17:34 GMT

There was slight error in what I posted.

UNIX95= ps -ef -o "pid, ppid, args, cpu" | sort -nk4 | tail -15

Also As Clay pointed also check timeslice kernel setting.

Anil

Re: %CPU 100% with %SYS > 70%

Andrew Cowan — Wed, 15 Jun 2005 02:15:07 GMT

I have not had time to look at the attachments but do you have anything like a modem process that is being backgrounded? I had a situation a while ago where a user was running "cu" over a modem port and then backgrounding it with "^Z". This drove the system nuts and produced a very similar symptom to that described here. The giveaway was that the interrupts per second climbed relentlessly.

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Wed, 15 Jun 2005 03:48:54 GMT

Hi RAC,

Attached find the output with the "tail" :)

Thks

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Wed, 15 Jun 2005 04:27:04 GMT

Hi,

The time slice is set to "10", I think it's the default.

Pedro

Re: %CPU 100% with %SYS > 70%

Pradeep_3 — Wed, 15 Jun 2005 04:50:24 GMT

Can you tell us is this situation is for whole day or in peak hour or is it for a particular time period.

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Wed, 15 Jun 2005 05:06:01 GMT

Hi,

The %SYS is allways very high with some peaks as posted before:

oot@poru9002:/stand# sar -u

HP-UX poru9002 B.11.00 U 9000/800 06/15/05

00:00:00 %usr %sys %wio %idle
00:20:00 10 28 1 62
00:40:00 11 29 1 59
01:00:00 13 31 3 54
01:20:00 13 30 7 50
01:40:00 9 28 1 62
02:00:00 13 31 3 54
02:20:01 9 28 1 62
02:40:00 9 28 1 62
03:00:00 13 30 3 53
03:20:00 9 28 1 62
03:40:00 11 28 1 60
04:00:00 16 32 5 48
04:20:00 12 29 1 58
04:40:00 15 29 2 53
05:00:00 15 32 3 51
05:20:00 20 31 2 47
05:40:01 55 38 4 3
06:00:00 51 43 3 4
06:20:00 46 35 3 17
06:40:00 42 39 1 18
07:00:00 23 39 4 34
07:20:00 10 28 1 61
07:40:00 10 28 2 59
08:00:01 16 32 6 46
08:20:00 16 29 3 51
08:40:00 15 30 3 51
09:00:00 19 33 3 45
09:20:00 19 31 2 48
09:40:01 25 31 7 37
10:00:00 39 36 11 15
10:20:00 37 31 8 24
10:40:00 13 29 3 55
11:00:01 37 31 5 27

Average 20 31 3 45

Re: %CPU 100% with %SYS > 70%

Eric Antunes — Wed, 15 Jun 2005 05:25:42 GMT

Hi Pedro,

What are this onl_rts017 processes?

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 9511 mpbe 241 20 30344K 4244K run 193:27 42.34 42.27 onl_rts027
1 ? 9207 mpbe 226 20 34276K 5432K run 188:52 30.76 30.71 onl_rts027

Can you kill them to see if they are the issue??

Best Regards,

Eric Antunes

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Wed, 15 Jun 2005 05:37:47 GMT

Hi Eric,

I don't think the problem is with onl_rts027 processes because if I kill them, the %CPU drops, but the %SYS remains higher than 40% with %USR very low...around 2%

Pedro

Re: %CPU 100% with %SYS > 70%

Tim Sanko — Wed, 15 Jun 2005 12:27:32 GMT

Pedro,

You might want to check shared memory with an ipcs. If too much shared memory is locked up it remains even through reboots. (such as oracle uses).

There may be other issues. Check for other processes that are eating up the CPU and see if you can locate any memory bottleneck (if there is one). If you are close to swapping, then it is worse if you start robbing the Disk Cache that if you are overloaded and just swap...

Glance works well for this. Otherwise you have to pull stuff out of the process tables with a C program to get what you would need.

Tim

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Thu, 16 Jun 2005 10:12:43 GMT

Hi Tim,

Here is the output of ipcs:

root@poru9002:/home/multiux# ipcs
IPC status from /dev/kmem as of Thu Jun 16 16:10:49 2005
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x3c1c024a -Rrw--w--w- root root
q 1 0x3e1c024a --rw-r--r-- root root
Shared Memory:
m 0 0x411c022a --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412005eb --rw-rw-rw- root root
m 31747 0x665bd3ec --rw-r----- oracle dba
m 15876 0x2142dbf0 --rw-rw---- oracle dba
m 44549 0x40340041 --rw-rw-rw- wsprg users
Semaphores:
s 0 0x411c022a --ra-ra-ra- root root
s 1 0x4e0c0002 --ra-ra-ra- root root
s 2 0x412005eb --ra-ra-ra- root root
s 3 0x01090522 --ra-r--r-- root root
s 4 0x411c3bd7 --ra-ra-ra- root root
s 5 0x61200a25 --ra-ra-ra- root root
s 6 0x73200a25 --ra-ra-ra- root root
s 7 0x70200a25 --ra-ra-ra- root root
s 8 0x69200a25 --ra-ra-ra- root root
s 9 0x75200a25 --ra-ra-ra- root root
s 10 0x63200a25 --ra-ra-ra- root root
s 11 0x64200a25 --ra-ra-ra- root root
s 12 0x66200a25 --ra-ra-ra- root root
s 13 0x6c200a25 --ra-ra-ra- root root
s 14 0x6d200a25 --ra-ra-ra- root root
s 15 0x6f200a25 --ra-ra-ra- root root
s 16 0x410c0384 --ra-ra-ra- root root
s 17 0x00000000 --ra-r----- oracle dba
s 530 0x52200a25 --ra-ra-ra- root root
s 51219 0x00000000 --ra-ra---- oracle dba
s 1541652 0x00340041 --ra-ra-ra- wsprg users
s 231957 0x0034000c --ra-ra-ra- wsprg users
s 3978774 0x00000000 --ra------- ws7mp users
s 463383 0x00000000 --ra------- ws7mp users
s 255000 0x00000000 --ra------- ws7mp users
s 74777 0x00000000 --ra------- mpliviu users
s 131610 0x00000000 --ra------- ws7mp users
s 334876 0x00000000 --ra------- mpliviu users
s 2118173 0x00000000 --ra------- mpliviu users
s 605214 0x00000000 --ra------- mpliviu users
s 469023 0x00000000 --ra------- multiux users
s 4297248 0x00000000 --ra------- mpliviu users
s 1118241 0x00000000 --ra------- mpliviu users
s 1574 0x00000000 --ra------- systemmp users
s 1067 0x00000000 --ra------- systemmp users
root@poru9002:/home/multiux#

Any ideia of what might be wrong? There is no swap...

Re: %CPU 100% with %SYS > 70%

Eric Antunes — Thu, 16 Jun 2005 10:20:59 GMT

Hi Pedro,

Can you post the following file:

#sar -v 6 20 > sar_v.log

Regards,

Eric Antunes

Re: %CPU 100% with %SYS > 70%

Fred Ruffet — Thu, 16 Jun 2005 10:29:54 GMT

What have you got in your syslog.log file ? Look for IO problems on some disks (EMS events, IO errors...)

Your don't swap, %wio is low (null), one disk have a wait average higher than others... Maybe a disk problem or a RAID rebuild.

Regards,

Fred

Re: %CPU 100% with %SYS > 70%

Eric Antunes — Thu, 16 Jun 2005 10:35:39 GMT

Hi again,

From ipcs, you have lots of Semaphores (compared with my Production System) but I can't tell you if this is the cause of your issue...

Regards,

Eric Antunes

Re: %CPU 100% with %SYS > 70%

Pedro Cirne — Thu, 16 Jun 2005 10:38:17 GMT

Hi Eric,

Attached find the sar statistics.

Another sar:

root@poru9002:/home# sar -u 5 10

HP-UX poru9002 B.11.00 U 9000/800 06/16/05

16:36:46 %usr %sys %wio %idle
16:36:51 18 82 0 0
16:36:56 23 77 0 0
16:37:01 18 82 0 0
16:37:06 24 76 0 0
16:37:11 19 81 0 0
16:37:16 23 77 0 0
16:37:21 22 78 0 0
16:37:26 22 78 0 0
16:37:31 21 79 0 0
16:37:36 18 82 0 0

Average 21 79 0 0

Thks

Re: %CPU 100% with %SYS > 70%

Eric Antunes — Thu, 16 Jun 2005 10:47:55 GMT

Hi,

Check the disks states and statuses:

#ioscan -fnCdisk

#iostat -t 6 20 > iostat_t.log

Eric