%CPU 100% with %SYS > 70%

[Pedro Cirne]

Hi,

I'm having a performance problem with an L2000-44. The CPU is allways at 100%, but the strange thing is that %SYS is allways greater than 70%. Any ideia of what the problem might be? Some statistics attatched :-)

Thank you!

[Pete Randall]

Are you swapping? Run "swapinfo -tam" and "vmstat 1 5" and post the output.


Pete

[RAC_1]

What are those mpbe processes and multiux processes??

Can you post following??

UNIX95= ps -ef -o "pid, ppid, args, cpu" | sort -nk4 | head -15

Anil

[A. Clay Stephenson]

You have a rather high value for process switches/second. I suspect that you have a number of small processes that are being spawned and executed. THe other thing that could cause this is a very low timeslice setting. What is your timeslice value?

[Pedro Cirne]

Hi,

Attahced find the statistics requested.

Thks

[RAC_1]

There was slight error in what I posted.

UNIX95= ps -ef -o "pid, ppid, args, cpu" | sort -nk4 | tail -15

Also As Clay pointed also check timeslice kernel setting.

Anil

[Andrew Cowan]

I have not had time to look at the attachments but do you have anything like a modem process that is being backgrounded? I had a situation a while ago where a user was running "cu" over a modem port and then backgrounding it with "^Z". This drove the system nuts and produced a very similar symptom to that described here. The giveaway was that the interrupts per second climbed relentlessly.

[Pedro Cirne]

Hi RAC,

Attached find the output with the "tail" :)

Thks

[Pedro Cirne]

Hi,

The time slice is set to "10", I think it's the default.

Pedro

[Pradeep_3]

Can you tell us is this situation is for whole day or in peak hour or is it for a particular time period.

[Pedro Cirne]

Hi,

The %SYS is allways very high with some peaks as posted before:

oot@poru9002:/stand# sar -u

HP-UX poru9002 B.11.00 U 9000/800 06/15/05

00:00:00 %usr %sys %wio %idle
00:20:00 10 28 1 62
00:40:00 11 29 1 59
01:00:00 13 31 3 54
01:20:00 13 30 7 50
01:40:00 9 28 1 62
02:00:00 13 31 3 54
02:20:01 9 28 1 62
02:40:00 9 28 1 62
03:00:00 13 30 3 53
03:20:00 9 28 1 62
03:40:00 11 28 1 60
04:00:00 16 32 5 48
04:20:00 12 29 1 58
04:40:00 15 29 2 53
05:00:00 15 32 3 51
05:20:00 20 31 2 47
05:40:01 55 38 4 3
06:00:00 51 43 3 4
06:20:00 46 35 3 17
06:40:00 42 39 1 18
07:00:00 23 39 4 34
07:20:00 10 28 1 61
07:40:00 10 28 2 59
08:00:01 16 32 6 46
08:20:00 16 29 3 51
08:40:00 15 30 3 51
09:00:00 19 33 3 45
09:20:00 19 31 2 48
09:40:01 25 31 7 37
10:00:00 39 36 11 15
10:20:00 37 31 8 24
10:40:00 13 29 3 55
11:00:01 37 31 5 27

Average 20 31 3 45

[Eric Antunes]

Hi Pedro,

What are this onl_rts017 processes?

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 9511 mpbe 241 20 30344K 4244K run 193:27 42.34 42.27 onl_rts027
1 ? 9207 mpbe 226 20 34276K 5432K run 188:52 30.76 30.71 onl_rts027

Can you kill them to see if they are the issue??

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi Eric,

I don't think the problem is with onl_rts027 processes because if I kill them, the %CPU drops, but the %SYS remains higher than 40% with %USR very low...around 2%

Pedro

[Tim Sanko]

Pedro,

You might want to check shared memory with an ipcs. If too much shared memory is locked up it remains even through reboots. (such as oracle uses).

There may be other issues. Check for other processes that are eating up the CPU and see if you can locate any memory bottleneck (if there is one). If you are close to swapping, then it is worse if you start robbing the Disk Cache that if you are overloaded and just swap...

Glance works well for this. Otherwise you have to pull stuff out of the process tables with a C program to get what you would need.

Tim

[Pedro Cirne]

Hi Tim,

Here is the output of ipcs:

root@poru9002:/home/multiux# ipcs
IPC status from /dev/kmem as of Thu Jun 16 16:10:49 2005
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x3c1c024a -Rrw--w--w- root root
q 1 0x3e1c024a --rw-r--r-- root root
Shared Memory:
m 0 0x411c022a --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412005eb --rw-rw-rw- root root
m 31747 0x665bd3ec --rw-r----- oracle dba
m 15876 0x2142dbf0 --rw-rw---- oracle dba
m 44549 0x40340041 --rw-rw-rw- wsprg users
Semaphores:
s 0 0x411c022a --ra-ra-ra- root root
s 1 0x4e0c0002 --ra-ra-ra- root root
s 2 0x412005eb --ra-ra-ra- root root
s 3 0x01090522 --ra-r--r-- root root
s 4 0x411c3bd7 --ra-ra-ra- root root
s 5 0x61200a25 --ra-ra-ra- root root
s 6 0x73200a25 --ra-ra-ra- root root
s 7 0x70200a25 --ra-ra-ra- root root
s 8 0x69200a25 --ra-ra-ra- root root
s 9 0x75200a25 --ra-ra-ra- root root
s 10 0x63200a25 --ra-ra-ra- root root
s 11 0x64200a25 --ra-ra-ra- root root
s 12 0x66200a25 --ra-ra-ra- root root
s 13 0x6c200a25 --ra-ra-ra- root root
s 14 0x6d200a25 --ra-ra-ra- root root
s 15 0x6f200a25 --ra-ra-ra- root root
s 16 0x410c0384 --ra-ra-ra- root root
s 17 0x00000000 --ra-r----- oracle dba
s 530 0x52200a25 --ra-ra-ra- root root
s 51219 0x00000000 --ra-ra---- oracle dba
s 1541652 0x00340041 --ra-ra-ra- wsprg users
s 231957 0x0034000c --ra-ra-ra- wsprg users
s 3978774 0x00000000 --ra------- ws7mp users
s 463383 0x00000000 --ra------- ws7mp users
s 255000 0x00000000 --ra------- ws7mp users
s 74777 0x00000000 --ra------- mpliviu users
s 131610 0x00000000 --ra------- ws7mp users
s 334876 0x00000000 --ra------- mpliviu users
s 2118173 0x00000000 --ra------- mpliviu users
s 605214 0x00000000 --ra------- mpliviu users
s 469023 0x00000000 --ra------- multiux users
s 4297248 0x00000000 --ra------- mpliviu users
s 1118241 0x00000000 --ra------- mpliviu users
s 1574 0x00000000 --ra------- systemmp users
s 1067 0x00000000 --ra------- systemmp users
root@poru9002:/home/multiux#

Any ideia of what might be wrong? There is no swap...

[Eric Antunes]

Hi Pedro,

Can you post the following file:

#sar -v 6 20 > sar_v.log

Regards,

Eric Antunes

[Fred Ruffet]

What have you got in your syslog.log file ? Look for IO problems on some disks (EMS events, IO errors...)

Your don't swap, %wio is low (null), one disk have a wait average higher than others... Maybe a disk problem or a RAID rebuild.

Regards,

Fred

[Eric Antunes]

Hi again,

From ipcs, you have lots of Semaphores (compared with my Production System) but I can't tell you if this is the cause of your issue...

Regards,

Eric Antunes

[Pedro Cirne]

Hi Eric,

Attached find the sar statistics.

Another sar:

root@poru9002:/home# sar -u 5 10

HP-UX poru9002 B.11.00 U 9000/800 06/16/05

16:36:46 %usr %sys %wio %idle
16:36:51 18 82 0 0
16:36:56 23 77 0 0
16:37:01 18 82 0 0
16:37:06 24 76 0 0
16:37:11 19 81 0 0
16:37:16 23 77 0 0
16:37:21 22 78 0 0
16:37:26 22 78 0 0
16:37:31 21 79 0 0
16:37:36 18 82 0 0

Average 21 79 0 0

Thks

[Eric Antunes]

Hi,

Check the disks states and statuses:

#ioscan -fnCdisk

#iostat -t 6 20 > iostat_t.log

Eric

[Pedro Cirne]

Hi Eric,

Attached find the ouput of iostat, I can't find anything wrong on ioscan, all the disk are CLAIMED:

root@poru9002:/var/adm/syslog# cat ioscan_Cdisk.out
Class I H/W Path Driver S/W State H/W Type Description
=================================================================================
disk 0 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST39103LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 1 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST39204LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 6x/32x
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 23 0/3/0/0.54.0.5.0.0.0 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t0d0 /dev/rdsk/c10t0d0
disk 24 0/3/0/0.54.0.5.0.6.3 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t6d3 /dev/rdsk/c10t6d3
disk 25 0/3/0/0.54.0.5.0.6.4 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t6d4 /dev/rdsk/c10t6d4
disk 26 0/3/0/0.54.0.5.0.6.5 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t6d5 /dev/rdsk/c10t6d5
disk 27 0/3/0/0.54.0.5.0.6.6 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t6d6 /dev/rdsk/c10t6d6
disk 28 0/3/0/0.54.0.5.0.6.7 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c10t6d7 /dev/rdsk/c10t6d7
disk 19 0/5/0/0.4.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c6t4d0 /dev/rdsk/c6t4d0
disk 20 0/5/0/0.5.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c6t5d0 /dev/rdsk/c6t5d0
disk 29 0/6/0/0.84.1.5.0.0.0 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t0d0 /dev/rdsk/c12t0d0
disk 30 0/6/0/0.84.1.5.0.6.3 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t6d3 /dev/rdsk/c12t6d3
disk 31 0/6/0/0.84.1.5.0.6.4 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t6d4 /dev/rdsk/c12t6d4
disk 32 0/6/0/0.84.1.5.0.6.5 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t6d5 /dev/rdsk/c12t6d5
disk 33 0/6/0/0.84.1.5.0.6.6 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t6d6 /dev/rdsk/c12t6d6
disk 34 0/6/0/0.84.1.5.0.6.7 sdisk CLAIMED DEVICE EMC SYMMETRIX
/dev/dsk/c12t6d7 /dev/rdsk/c12t6d7

Thks

[Eric Antunes]

Hi Pedro,

You have a good ioscan but your iostat report seems strange because of low activity and tout of -0: I'll investigate this further.

Meanwhile, I found a thread about high %sys and it may be interesting for you:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=825115

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi Eric,

Do you think the kernel parameter max_fcp_reqs set to 128 may be the cause of this high SYS%?

The default is 512, but in the past this server was connected to an old storage that required this max_fcp_reqs to be 128.

Now it's connected to a new Symmetrix DMX-800 so probably this may me the cause, but I'm not sure...

Thks

[Eric Antunes]

Hi Pedro,

Yes, I think you should use the default 512 value again since you have a new adapter.

Best Regards,

Eric Antunes

[Pedro Cirne]

Hi,

But I don't think this is the root cause of this CPU bottleneck on %SYS...

Pedro