topic Re: High CPU/Sys utilization in Operating System - HP-UX

High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:11:20 GMT

Got a N-4000, 4 440CPU's, 6GB Ram, DBC set to 2/7max. Glance constantly reports "CPU_Bottleneck is 100%" system does have high memory/swap/network or disk i/o contention. Syscalls Rate anywhere from 25,000 to a high of 86K,confirmed with vmstat, interupts at the cpu at 6179, with load averages of 1.9 to 2.5. vmstat shows NO sr(scan rate). TOP and trending software shows continued high USR/SYS CPU utilization.

Re: High CPU/Sys utilization

Ian Dennison_1 — Mon, 03 Feb 2003 17:16:32 GMT

How much system cpu %? What does 'sar -u' tell you for '%wio'?

What applications are your running on this system? Any recent changes? is the nature of the Application to work in short bursts or more prolonged periods?

System is handling processes and memory management, so check how much memory you are using and how many active processes you have.

Any paging out shown in 'vmstat'? Likewise, deactivations?

Share and Enjoy! Ian

Re: High CPU/Sys utilization

Bill McNAMARA_1 — Mon, 03 Feb 2003 17:17:09 GMT

attach the sar.report of the following:

sar -o /tmp/sarfile 2 1000
2 = interval between sar collection
1000 = number of samples to record
=> total time = 2000 seconds (~30 minutes)
sar -Af /tmp/sarfile > /tmp/sar.report

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:20:41 GMT

I have attached a cpu and mem utilization from glance plus. Any ideas?

Re: High CPU/Sys utilization

Bill McNAMARA_1 — Mon, 03 Feb 2003 17:22:42 GMT

what processes have been niced?

Re: High CPU/Sys utilization

Jose Mosquera — Mon, 03 Feb 2003 17:23:26 GMT

Hi,

Pls check the **swapper** presence on top 1st page, if be there your system handles activation and reactivation of processes when free memory falls bellow minfree, or when the system appears to be ???trashing???.

dbc_max_pct Kernel parameter fix this problem, the recomendations are from 5% to 10% depends on the total amount of memory in your system.

The minimun value must be 300MB, then you can try to put a low percentage, I sugest you that first try with 25% and look at the backup performance in order to know if this is a good percentage. After you do this and reboot your dont want to see the USED line for the device lines from swapinfo -mt > 0. This means no memory pressure and no paging. This will increase performance considerably.

Rgds.

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:23:33 GMT

Im running the sar -o now.. I have glance plus by the way. Im really thinking that this system is overloaded and needs addtional processors. I have 9 oracle instances with 3 instance of Peoplesoft running but they are mostly idle. We dev/test/UAT etc.. with these instance but never utilize more than 2 at a time.

Re: High CPU/Sys utilization

Ian Dennison_1 — Mon, 03 Feb 2003 17:48:39 GMT

It sounds like starting the third database is robbing you of resources.

How many processes do you have running on the system (ps -ef |wc -l). How close it is to the nproc value?

Peoplesoft should have a set of recommended values for the kernel for a single installation in its install doc. It would pay to check these against the current config, as you may not have enough resources for 3.

Share and Enjoy! Ian

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:52:24 GMT

ALL- MISTAKE(TYPEO) SYSTEM DOES NOT HAVE HIGH MEM/SWAP/NETWORK or disk IO Contention, JUST HIGH CPU..

Sorry

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:55:46 GMT

here is the sar.report

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 17:57:54 GMT

Only 318 processes, nproc 16k

Re: High CPU/Sys utilization

Ian Dennison_1 — Mon, 03 Feb 2003 18:10:05 GMT

Can we try to look at Glance to see what it detects?

Option 'c' will show CPU report by State - look for Contect Switches or Interrupts. Also, what appears high in the list of processes using top?

A real obvious one is 'timeslice' being 1 instead of 10, but if you have not had this performance problem before, this is not a likely cause.

Share and Enjoy! Ian

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 18:24:11 GMT

does processes that run for example: user=oracle executes /bin/ksh /oraunix/home/db_check.sh, does this show up as system time since the /bin/ksh shell was called 1st?

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 18:30:12 GMT

here is glance with c flag, This shows the high cpu but low mem/swap etc..

ata Collected: 02/03/03 13:05:06

Refresh Interval: 5 seconds

GlancePlus Started/Reset: 02/03/03 13:02:23
B3692A GlancePlus C.02.40.00 13:05:11 msuapd07 9000/800 Current Avg High
-------------------------------------------------------------------------------
CPU Util SSSSSSSSSSSSSNNNNNNNNNNNNNNNNNUUUUUUUUUUU | 80% 86% 95%
Disk Util FFFFFFFFFFFFF | 25% 23% 41%
Mem Util SSSUUUUUUUUUUUUUUUUUUUUUBBBB | 54% 54% 54%
Swap Util UUUUUUURRRRRRRRRRR | 35% 35% 35%
-------------------------------------------------------------------------------
CPU REPORT Users= 5
State Current Average High Time Cum Time
--------------------------------------------------------------------------------
User 21.5 14.9 31.5 1.13 25.27
Nice 33.0 40.0 52.9 1.73 67.67
Negative Nice 0.4 0.2 1.9 0.02 0.36
RealTime 0.2 0.4 2.7 0.01 0.68
System 21.3 26.3 51.5 1.12 44.40
Interrupt 0.4 0.6 1.5 0.02 1.03
ContextSwitch 0.2 0.2 0.4 0.01 0.41
Traps 0.0 0.0 0.2 0.00 0.05
Vfaults 3.2 3.1 6.8 0.17 5.24
Idle 19.8 14.2 28.4 1.04 24.00

Top CPU user: PID 15671, ksh 43.2% cpu util
Page 1 of 2

Re: High CPU/Sys utilization

Roland Rebstock_1 — Mon, 03 Feb 2003 18:57:58 GMT

Re: High CPU/Sys utilization

Sridhar Bhaskarla — Mon, 03 Feb 2003 19:12:29 GMT

Hi Roland,

1. There are some processes that are running with a high nice value (low priority). You may want to get them fixed unless you intend to run them that way.

UNIX95= ps -e -o "nice args" |sort -n

The first column should be 20. If it is above 20, then the process is in nice state. Use renice command to change the nice value. If any process is started through cron or in background, they will be started with a high nice value.

2. Looks like you are running a lot of scripts. And there is this "find" command. Find command can be lethal on the system if it is done starting from /. You may want to identify these scripts, see what they do and kill the ones that are not needed. Don't run any script that does "find" during the business hours.

-Sri