help: unusually high system load vs. user load. how to debug/fix?

[Edwin R. Rivera_1]

hello all,

i'm working on a hp-ux 9000 v11.0 (64bit) server with 5 cpu's. the current load avg on the box is 1.67 (5min avg). however, when i watch a top session, i notice that the system loads are unusually high for this server and was wondering if hp-ux had a way of finding the culprit other than just my usual "ps -ef" and basic unix tools?

i've attached an example of the "top" summary for the cpu's.

thanks in advance everyone.

[harry d brown jr]

Yes, with glance/measureware it's easy. It costs a little, but you can install a trial version for free (30 day limit). With glance you can "drill" down into the process to see what it is doing, what files it has open, what it's memory demands are, etc... Plus the data is collected in real time, unlike top, sar, vmstat, and iostat.

live free or die
harry

[Helen French]

Hi Edwin,

The best option is to use Glanceplus software from HP. It will defenitely help you to find out each and every thing you needed on this issue. This is not a freeware, so you have to purchase it from HP ( you will get a free trial copy though ). Tha administration is also easy.

Check www.docs.hp.com for more information on how Glance works ! You can also check other options like perfview, sar, vmstat, netstat, PRM etc.

HTH,
Shiju

[Steve Steel]

Hi


Can only concur glance since your machine is well occupied.

Check your syslog and swapinfo -ta

You may need to free some memory by reducing dbc_max_pct from its default

see kmtune -l

A quick and dirty one is

UNIX95= ps -e -o pcpu -o ruser -o args|sort -nr|grep -v %CPU|head -10

This will give you the top 10 cpu users.


Steve Steel

[Bill Hassell]

There's nothing wrong with a high CPU load. After all, isn't that what you bought the computer to do? The question then becomes: are the programs wasting CPU cycles by doing things inefficiently? That of corse can only be fixed by rewriting the programs. This CPU load may also be caused by processes that were trashed when a remote user (using a PC that mysteriously rebooted) dropped their connection and the process(es) are orpahns that should be terminated.

If rewriting the programs is not an option, then replacing your CPU's with higher speed CPU's may improve things.

However, your top list shows a mostly idle machine with higher than normal system overhead. This overhead may indeed be due to orphan processes or programs that do a lot of polling, perhaps on the LAN.

[Alan Riggs]

Glance is a fine tool, but may not be necessary in this case. You didn't include the details of your top command, but the system load is just an aggregate of individual system processes which will register in top. (Remember also that the "system" versus "user" designations are not always clear-cut or intuitively obvious.)

Look at your top CPU users in detail and see which ones are system processes. Just identifying which processes are running high might give you a good idea of the problem.

And, of course, the fact that you still have ample idel time across all CPUs means whatever is going on is unlikely to be causing any CPU based performance issues.

[harry d brown jr]

Edwin, are you running EMS on this server? Also, as I believe Alan asked, could you attach a top output?

live free or die
harry

[Martha Mueller]

I know this is kind of late, but....
We had a very similar situation when we upgraded our version of sybase from 11.03 to 11.9.2. Turns out that the optimizer had changed and so we had table scans going on. There really wasn't any way to tell from the operating system, except for the fact that there was high system load vs user load. We ended up having to set some flags within sybase and had to re-write some code.

Good luck.