CPU Performance

[Nobody's Hero]

I have an K570 with 4 processors. The box get taxed a lot from Oracle developers. My question is, when I'm in glance and look at the cpu graph. Nearly 80% of all processors are reading blue meaning sys. Is this always a bad thing?

Thanks,
Bob

[Alexander M. Ermes]

Hi there.
If the sys part of that graph is that big, the system is mostly busy with admin processes for itself.
Check the disk i/o, lan load etc.
Check the global process list, what process is eating your resources. It might be a dead Oracle process.
Rgds
Alexander M. ERmes

[Rita C Workman · Accepted solution]

Robert,

If I have read this post correctly..than YES this is NOT GOOD !
You would want your users to have more of the system to run their work...

My first question would be what is dbc_max_pct and dbc_min_pct set at? If you left it at the default value, than more than likely it is way too high (default=50 for max)!. That means the system is taking 50% off the top. Most systems will run with max=5 to 15%

But this is just someplace to start....

Rgrds,
Rita

[Krishna Prasad]

What is your overall % of cpu usage? Is it 80% and all of the 80% is sys?

What is the percentage per cpu and what is the load per cpu? In glance hit the 'a' key and you can see this.

If you are 80% of cpu and it is the system using the cpu's what is % memory and %disk swap%?

If % of memory is also high hit 'm' in glance and see how many page out's and deactivations,reactivations you have if any.

Also, look at % swap space used.

You should also look at disk I/O hit 'i' or 'd' in glance and check for bottlenecks.

[Tim D Fulford]

As everyone has said above too much sys is generally a bad thing. you need to determine what is taking it up. You can look in glance & check out the big hitters & select individual processes, even using top could help.

If you mave measure Ware running you can look at what processes (pids) are consuming syscalls

here is the reptall file I would use
REPORT "MWA Export !DATE !TIME Logfile: !LOGFILE !COLLECTOR !SYSTEM_ID"
FORMAT ASCII
HEADINGS ON
SEPARATOR="|"
SUMMARY=60
MISSING=0
DATA TYPE PROCESS
DATE
TIME
PROC_PROC_ID
PROC_PROC_NAME
PROC_PARENT_PROC_ID
PROC_PRI
PROC_STOP_REASON
PROC_CPU_TOTAL_UTIL
PROC_CPU_SYS_MODE_UTIL
PROC_DISK_PHYS_IO_RATE
PROC_IO_BYTE_RATE
PROC_DISK_SYSTEM_IO_RATE
PROC_PRI_WAIT_PCT
PROC_DISK_SUBSYSTEM_WAIT_PCT
PROC_SEM_WAIT_PCT
PROC_OTHER_IO_WAIT_PCT
PROC_LAN_WAIT_PCT
PROC_SLEEP_WAIT_PCT
PROC_MEM_WAIT_PCT
PROC_NFS_WAIT_PCT
PROC_IPC_SUBSYSTEM_WAIT_PCT
PROC_SYS_WAIT_PCT

Now run an extract for the processes
# extract -xp -v -p -r -b -e
this will output to xfrdPROCESS.asc

I would then take a look at the processes that appear the most (unless you have a known dodggy pid, then move onto the last cmd)
# awk -F"|" 'NR>3{print $4}' xfrdPROCESS.asc | sort | uniq -c | sort -n

This will do a counted sort of the process names. To get the PID do

# awk -F"|" '$4~PNAME{print $3, $4}' PNAME= xfrdPROCESS.asc | sort | uniq -c | sort -n
This will show what PIS's are attached to the process names.

One you have the PID(s) that run the most do
# awk -F"|" 'NR<3 || $3==""{print $0}' xfrdPROCESS.asc > outfile..asc

Import outfile..asc into excel & draw pritty graphs for each process-pid. Look out for high sys CPU % (compared to CPU %), look for any % wait system values.

Basically high system cpu could be due to VERY HIGH IO rates or programmers/developpers making excessive system calls.

The other thing to check for, are your developers writting code that trigger large sequential scans of tables?

The problem you outline could have many sources, I hope that I've given you some ideas on how to tackle the problem.

good luck

Tim

[A. Clay Stephenson]

Hi Robert:

Since you mentioned Oracle, one of the first things to check is the value of timeslice. One of the really dumb tuned parameter sets sets it to 1 and that would cause exactly the kind of CPU usage pattern that you are seeing due to very high context switching. It should have been left at 10. I would check this first thing.

[Chris Vail]

There are lots of ways to find out which process is using the most cpu cycles. There is glance plus, top and a host of other tools out there. However, I usually use the following command line:
ps -ef|sort -k 4nr,4|pg

This sorts the output of the ps command by the amount of CPU utilization. The highest numbers will be at the top. This quickly tells you where to begin looking to speed things up.

Give it a try
Chris