High SYS CPU Usage!

[Chris J. Solarczyk]

Hi All,

I'm trying to figure out an N class here that's taking a big CPU hit. The CPU utilization is almost always 100% with especially high SYS value (20-40%). The N class has 4 (440MHz) CPUs, 6 GB RAM, and is connected to an EMC Symm via fibre. Load averages are also high (7-10) and performance seems weak. I recently changed the Buffer Cache from 2 - 20% dynamic to 300 MB fixed because the system always used all 20% (1.2 GB). I thought this would help performance but it really didn't. I'm not swapping and disk/network/memory performance seems OK. Would adding a couple more CPU's help? How about increasing the Buffer Cache to perhaps 600 MB? Any suggestions would really be helpful.

Thanks

Chris

P.S. I attached a screen shot from Glance to illustrate.

[Rita C Workman]

Didn't see the attached file...but considering the last time I opened up an attachment I got virused.
Might I suggest if your dynamic buffer was set to 20% that you might want to change this parm setting dbc_max_pct from 20 to 10 and change dbc_min_pct to 5. These control how much the system is going to reserve off the top...and 20% of of 6gb is ..Ouch !! I don't know what you run on that box..or what your other settings are, but maybe this is a start for you

Hope it helps,
/rcw

[Carlo Henrico_1]

This may seem weird, however I was having similar symptoms some time ago and it eventually worked out to be a faulty SCSI array controller. The latest patch was loaded and the array controller was replaced and all my problems disappeared. The weird part was that nothing was being reported in any logs that the controller was faulty.

[Brian Hackley]

Chris,
User space CPU hogs can be identified with top. Finding System CPU hog culprit requires more extensive HPUX Detective work. There are various spots in GLANCE that can be helpful For example, check to see what syscalls are most frequest. fork and exec are two of the most expensive calls. Verify by looking at context switch CPU utilitization. Another expensive area is Spinlock (SWAIT) and Semaphore (BLOCK), check those two with top.

If all else fails, sometimes you need to do some kernel profiling to find out what code/routines in the kernel is/are being used the most which can point you to some tunables that might need to be changed. There are some tools the HP Support folks can make available to do this profiling if that become absolutely necessary.
Hope this helps,
-> Brian Hackley

[Chris J. Solarczyk]

Solution: The reason for the very high SYS CPU usage (and poor system performance) was a large number of mini-runaway "sh" processes. I say "mini" because each process had a CPU Util of 7-10 but Disk I/O Rate was 0 (via glance). After I killed all of these "sh" processes, they SYS CPU average went down to 10% and load dropped from 11 to 2! The server is again running like a champ. I believe that the sh procs got created when users exit'ed the application abnormally (i.e. click [X] on the emulator window) but I'm still researching.

[Neil Harris]

We had a similar problem on a K class this turned out to be a process running within a Universe data base where a user had started a big query and the exuted by turning off the terminal. It took considerable time to locate the problem as the application was not obvious in the normal ways of looking. Effectively this was a zombied process and required killing the problem was solved by killing universe. I wonder if you have a similar problem look for processes that are hidden.

[Hugh Trevelyan]

Chris,
I've had this before and if you're interested, writing a short script or a one liner can help immensley, but you sort of have to know where to look first:

ps -ef | grep $(date|awk'print $4}' | cut -d: -f1,2)