High CPU/Sys utilization

[Roland Rebstock_1]

Got a N-4000, 4 440CPU's, 6GB Ram, DBC set to 2/7max. Glance constantly reports "CPU_Bottleneck is 100%" system does have high memory/swap/network or disk i/o contention. Syscalls Rate anywhere from 25,000 to a high of 86K,confirmed with vmstat, interupts at the cpu at 6179, with load averages of 1.9 to 2.5. vmstat shows NO sr(scan rate). TOP and trending software shows continued high USR/SYS CPU utilization.

[Ian Dennison_1]

How much system cpu %? What does 'sar -u' tell you for '%wio'?

What applications are your running on this system? Any recent changes? is the nature of the Application to work in short bursts or more prolonged periods?

System is handling processes and memory management, so check how much memory you are using and how many active processes you have.

Any paging out shown in 'vmstat'? Likewise, deactivations?

Share and Enjoy! Ian

[Bill McNAMARA_1]

attach the sar.report of the following:

sar -o /tmp/sarfile 2 1000
2 = interval between sar collection
1000 = number of samples to record
=> total time = 2000 seconds (~30 minutes)
sar -Af /tmp/sarfile > /tmp/sar.report

[Roland Rebstock_1]

I have attached a cpu and mem utilization from glance plus. Any ideas?

[Bill McNAMARA_1]

what processes have been niced?

[Jose Mosquera]

Hi,

Pls check the **swapper** presence on top 1st page, if be there your system handles activation and reactivation of processes when free memory falls bellow minfree, or when the system appears to be ???trashing???.

dbc_max_pct Kernel parameter fix this problem, the recomendations are from 5% to 10% depends on the total amount of memory in your system.

The minimun value must be 300MB, then you can try to put a low percentage, I sugest you that first try with 25% and look at the backup performance in order to know if this is a good percentage. After you do this and reboot your dont want to see the USED line for the device lines from swapinfo -mt > 0. This means no memory pressure and no paging. This will increase performance considerably.

Rgds.

[Roland Rebstock_1]

Im running the sar -o now.. I have glance plus by the way. Im really thinking that this system is overloaded and needs addtional processors. I have 9 oracle instances with 3 instance of Peoplesoft running but they are mostly idle. We dev/test/UAT etc.. with these instance but never utilize more than 2 at a time.

[Ian Dennison_1]

It sounds like starting the third database is robbing you of resources.

How many processes do you have running on the system (ps -ef |wc -l). How close it is to the nproc value?

Peoplesoft should have a set of recommended values for the kernel for a single installation in its install doc. It would pay to check these against the current config, as you may not have enough resources for 3.

Share and Enjoy! Ian

[Roland Rebstock_1]

ALL- MISTAKE(TYPEO) SYSTEM DOES NOT HAVE HIGH MEM/SWAP/NETWORK or disk IO Contention, JUST HIGH CPU..

Sorry

[Roland Rebstock_1]

here is the sar.report

[Roland Rebstock_1]

Only 318 processes, nproc 16k

[Ian Dennison_1]

Can we try to look at Glance to see what it detects?

Option 'c' will show CPU report by State - look for Contect Switches or Interrupts. Also, what appears high in the list of processes using top?

A real obvious one is 'timeslice' being 1 instead of 10, but if you have not had this performance problem before, this is not a likely cause.

Share and Enjoy! Ian

[Roland Rebstock_1]

does processes that run for example: user=oracle executes /bin/ksh /oraunix/home/db_check.sh, does this show up as system time since the /bin/ksh shell was called 1st?

[Roland Rebstock_1]

here is glance with c flag, This shows the high cpu but low mem/swap etc..

ata Collected: 02/03/03 13:05:06

Refresh Interval: 5 seconds

GlancePlus Started/Reset: 02/03/03 13:02:23
B3692A GlancePlus C.02.40.00 13:05:11 msuapd07 9000/800 Current Avg High
-------------------------------------------------------------------------------
CPU Util SSSSSSSSSSSSSNNNNNNNNNNNNNNNNNUUUUUUUUUUU | 80% 86% 95%
Disk Util FFFFFFFFFFFFF | 25% 23% 41%
Mem Util SSSUUUUUUUUUUUUUUUUUUUUUBBBB | 54% 54% 54%
Swap Util UUUUUUURRRRRRRRRRR | 35% 35% 35%
-------------------------------------------------------------------------------
CPU REPORT Users= 5
State Current Average High Time Cum Time
--------------------------------------------------------------------------------
User 21.5 14.9 31.5 1.13 25.27
Nice 33.0 40.0 52.9 1.73 67.67
Negative Nice 0.4 0.2 1.9 0.02 0.36
RealTime 0.2 0.4 2.7 0.01 0.68
System 21.3 26.3 51.5 1.12 44.40
Interrupt 0.4 0.6 1.5 0.02 1.03
ContextSwitch 0.2 0.2 0.4 0.01 0.41
Traps 0.0 0.0 0.2 0.00 0.05
Vfaults 3.2 3.1 6.8 0.17 5.24
Idle 19.8 14.2 28.4 1.04 24.00


Top CPU user: PID 15671, ksh 43.2% cpu util
Page 1 of 2

[Sridhar Bhaskarla]

Hi Roland,

1. There are some processes that are running with a high nice value (low priority). You may want to get them fixed unless you intend to run them that way.

UNIX95= ps -e -o "nice args" |sort -n

The first column should be 20. If it is above 20, then the process is in nice state. Use renice command to change the nice value. If any process is started through cron or in background, they will be started with a high nice value.

2. Looks like you are running a lot of scripts. And there is this "find" command. Find command can be lethal on the system if it is done starting from /. You may want to identify these scripts, see what they do and kill the ones that are not needed. Don't run any script that does "find" during the business hours.

-Sri