Very high CPU utlization

[Sreekanth Makam]

Hi folks,
I doing some load test by applying 1000 users and each user will do some activity on server.
My server is RP8400 server which has 16 CPUs.

I alloted 9 CPUS to informix. 6 CPUs to my applications. and 1(0th CPU) to system. This is done using psets.

All CPUs are going 99%.

My questions is,, is it beacause of pset confugartion??...

Any body can just explain..that PSET will really help us..Or if it helps..how it helps? In which condition i must use pset.
With out pset configartion all cpus are going to 90%.(system= 40%, user = 50%)

And one more thing is...If IDLE =0% meance then it is good or bad.. Beacuse some says all cpus are used correclty and some says it must not go to 0%.

Please help me

[KapilRaj · Accepted solution]

if cpuidle=0
then
if runq > number of processors
then
echo Hey I am running out of cpu's
else
echo Hey no problem
fi
else
echo Hey no probs ..
fi

Kaps

[Trond Haugen]

As Kapil said "it depends"... 0% idle is it self no problem. But an indication that it MIGHT be.
Be sure to load the apropreate pset patches and at least the June 2001 Quality Pack.

Regards,
Trond

[Bill Hassell]

100% CPU usage is a good thing! After all, you paid a lot for the computers to do a lot of work. If you stop all the applications, I'll bet that the CPU usage will be 0%. In other words, the applcations are using the CPU. Now whether these applications are doing reasonable things and not wasting CPU cycles is not something the operating system can fix.

Allocating CPUs has always shown to be a waste of effort. HP-UX handles CPU resources very efficiently and most attempts to improve on this with manual assignment seldom results in improvements. NOTE: switching a program from one processor to another has ZERO overhead. When a context switch occurs, the opsystem has to select a CPU. There's virtually no difference in the steps needed to assign the CPU. No change occurs in memory--just a program counter and register setup is required, and it's required for dedicated CPUs too.

Now 40% system overhead is quite high for typical database systems and indicates that the programs are generating very high rates of system calls. Classic Unix tools are ill-equipped to measure and present the details needed to identify application problems. With a system this big, get a copy of GlancePlus as soon as possible so you can see the performance issues easily.

[Sreekanth Makam]

Yes,
u are all correct.. runq=4-5. so i have no prob in cpus..

Ok..As some said psets are not really needed in HPUX 11 but ..By psets only i got good performance. So iam continuing..

I used profile and tusk..it is also saying there are too many system call as some body told.

OK..Now my prob is how to make sys mode very less. Is there any kenal parameter that i can tune...Or it's all depended on the code i wrote???

[KapilRaj]

If possible re-write using less system calls and compare it with time (or is it timex) command

time

Regds,

Kaps

[Sreekanth Makam]

Hi kaps,

I don't want to change the code. I want to decrease the sys mode utlization. Is there any way. In hardware part..or anything..

Thnks in adv,
Sree

[Jeff Schussele]

Hi,

I'd start with a thorough review of your kernel parameters. Particularly the timeslice parm.
IF it's set to less than 10 that could cause a lot of unnecessary context switching & needless overhead. Set it to 10 if it's not already there. Also check the dbc_max/min_pct values.
No need to have the OS cache the Informix reads/writes if the DB is already doing it.
Handling a large buffer cache can put quite the load on a system. We've found that between 400 & 800 MB tends to be a sweet spot for buffer cache size. There are other parms that can affect system perf - so a thorough review is required here.

Rgds,
Jeff

[Sridhar Bhaskarla]

Hi Sreekanth,

You would use psets if you want to bind processes to only certain number of CPUs. It may be required when you have few processes on the system that are monopolizing the CPUS.

It's very good if the CPU is running at 90-95% and *really* bad if they are running constantly at 100%. You would know why if you support production systems. It's just a matter of 5%-10% CPU and that allows you to plan for future. I may save a little money if I keep my systems at 100% but lose a lot if my production is affected.

I am not sure if it is because of pset configuration. PSETs are useful if the process groups are independent. For ex., you have two applications and they don't interact with each other. The performance will degrade if you have processes that are dependent on the processes in a different pset.

So, try without psets and see how the system responds. Also, look at HP's itrc site to see if there are any patches for PSETs.

Last but not the least, code is the major issue in most of the issues. It can make a lot of difference.

-Sri

[Steve Lewis]

Informix is heavy on the following calls:
select()
read()
write()
semctl()

The more i/o you do, to the disk or the network, the more read/write calls you will do.
If you can offload some of the database connections from tcp socket onto shared memory/ipc you will reduce the calls.

If you are using KAIO, your system time will increase. Without KAIO and with AIOVPs, your system % reduces, but user % increases.

Having cpus at 99% is commonly a symptom of not using database indexes efficiently, doing sequential scans. Have you reviewed the indexes and updated the statistics recently?