High CPU utilization, what to look for?

[Jeff Patrick]

6 of my HP9000 servers, each identical, all have 2 identical processors, and all hardware in them is the same. All kernel parameters are identical on the 6 as well. Each server runs Oracle, 2 servers with about 50 users, the rest with about 40. On one of the 50-user servers, my CPU utilization throughout the day hovers in the 90-100% range, while on the other servers, it usually never tops 50%. I get approximately 15-20 notifications that CPU utilization has hit 99% from my monitoring software (Platinum Servervision) per day on this server, and none on any of the other 5. Nothing appears to be different from an HP-UX standpoint between these servers, and no Oracle configuration differences jump out at me either, yet I cannot figure out why this one server seems to be so much more heavily hit from a CPU standpoint. It has a few more users, but seemingly not enough to compensate for the doubling in CPU utilization over the other servers. Any suggestions, or things I can look for, from HP-UX or Oracle? Thanks!

Jeff

[James R. Ferguson]

Hi, Jeff:

You don't indicate whether the high CPU utilization is accompanied by a relatively high run-queue depth or not. You also need to determine whether the high activity is a user or system component. Given the environment you describe, it could simply be a particular user doing a particular query.

Various tools like 'sar', 'vmstat', 'glance' [in particular!], 'top' and 'uptime' are very helpful here.

I would suggest you step through the guidelines in documents #S3100002312A, #S3100002312B and #S3100002312C ("Sys Adm: determining the cause of system performance problems") to begin to get a clear picture.

'glance' is available for a trial basis on the Application CDs. It is otherwise as chargeable licensed product, but very worth the cost.

...JRF...

[Curtis Larson]

you could start with what process have a high cpu time:

ps -e | grep -v TIME | awk '{for (i=1;i<=NF;i++) if ( i != 2 ) printf("%10s ",$i);print "";}'| sort -rk 2,2 | head

that will give you the top cpu total time users. and if any of these have a parent process id of 1 that would be something to look at.

see what are the top cpu process currently running via top or glance. see what the biggest users are performing.

[Tim Malnati · Accepted solution]

I wonder if you have a user on that machine that likes to crunch reports a lot. I can't say that I'm familiar with the Platinum monitoring and performance tools, but I would suggest that the product must have some sort of method to drill down to the numbers. I've heard that Platinum has another product called dbvision that may be more in line with what your attempting to investigate. My suggestion is that you give Platinum support a call for some guidance. Maybe they have a demo arangement available for dbvision as well.

[Patrick Wallek]

When you see the high CPU utilization, the first thing I'd do is start 'top' and see what process(es) are at the top of the list. I suspect, as one of the other respondents suggested, that it is a user that is doing some sort of massive query, or report or something. It could be on purpose, or it could be a result of them not being familiar enough with the application.

Start with top, find the process, then find the user, and go from there.

[Shannon Petry]

As someone else mentioned, I would start with top. It's simple and painless (except for adding another load to the CPU....but all monitors do).

Even though all 6 systems are running oracle, there are probably different databases on each, with different sizes, different indexes, different queries, and perhaps different software accessing Oracle.
I'm working on a K460 (4CPU's, 8GB RAM, 8GB swap) now with 70 users averaging 90% CPU. The culprit is the way users are accesing oracle (through MS ODBC drivers...)

If oracle is the problem, it will be immediately apparent with TOP that oracle is eating the CPU. If it is not oracle, then use vmstat, top and sar if your cheap to find out what is killing you. If you have a budgett, invest in Glance! Awsome program well worth the $$ :)

Regards,
Shannon