cancel
Showing results for 
Search instead for 
Did you mean: 

Very high CPU utlization

SOLVED
Go to solution
Sreekanth Makam
Occasional Advisor

Very high CPU utlization

Hi folks,
I doing some load test by applying 1000 users and each user will do some activity on server.
My server is RP8400 server which has 16 CPUs.

I alloted 9 CPUS to informix. 6 CPUs to my applications. and 1(0th CPU) to system. This is done using psets.

All CPUs are going 99%.

My questions is,, is it beacause of pset confugartion??...

Any body can just explain..that PSET will really help us..Or if it helps..how it helps? In which condition i must use pset.
With out pset configartion all cpus are going to 90%.(system= 40%, user = 50%)

And one more thing is...If IDLE =0% meance then it is good or bad.. Beacuse some says all cpus are used correclty and some says it must not go to 0%.

Please help me
*** Let us share the knowledge ***
16 REPLIES
KapilRaj
Honored Contributor
Solution

Re: Very high CPU utlization

if cpuidle=0
then
if runq > number of processors
then
echo Hey I am running out of cpu's
else
echo Hey no problem
fi
else
echo Hey no probs ..
fi

Kaps
Nothing is impossible
Trond Haugen
Honored Contributor

Re: Very high CPU utlization

As Kapil said "it depends"... 0% idle is it self no problem. But an indication that it MIGHT be.
Be sure to load the apropreate pset patches and at least the June 2001 Quality Pack.

Regards,
Trond
Bill Hassell
Honored Contributor

Re: Very high CPU utlization

100% CPU usage is a good thing! After all, you paid a lot for the computers to do a lot of work. If you stop all the applications, I'll bet that the CPU usage will be 0%. In other words, the applcations are using the CPU. Now whether these applications are doing reasonable things and not wasting CPU cycles is not something the operating system can fix.

Allocating CPUs has always shown to be a waste of effort. HP-UX handles CPU resources very efficiently and most attempts to improve on this with manual assignment seldom results in improvements. NOTE: switching a program from one processor to another has ZERO overhead. When a context switch occurs, the opsystem has to select a CPU. There's virtually no difference in the steps needed to assign the CPU. No change occurs in memory--just a program counter and register setup is required, and it's required for dedicated CPUs too.

Now 40% system overhead is quite high for typical database systems and indicates that the programs are generating very high rates of system calls. Classic Unix tools are ill-equipped to measure and present the details needed to identify application problems. With a system this big, get a copy of GlancePlus as soon as possible so you can see the performance issues easily.


Bill Hassell, sysadmin
Sreekanth Makam
Occasional Advisor

Re: Very high CPU utlization

Yes,
u are all correct.. runq=4-5. so i have no prob in cpus..

Ok..As some said psets are not really needed in HPUX 11 but ..By psets only i got good performance. So iam continuing..

I used profile and tusk..it is also saying there are too many system call as some body told.

OK..Now my prob is how to make sys mode very less. Is there any kenal parameter that i can tune...Or it's all depended on the code i wrote???




*** Let us share the knowledge ***
KapilRaj
Honored Contributor

Re: Very high CPU utlization

If possible re-write using less system calls and compare it with time (or is it timex) command

time

Regds,

Kaps
Nothing is impossible
Sreekanth Makam
Occasional Advisor

Re: Very high CPU utlization

Hi kaps,

I don't want to change the code. I want to decrease the sys mode utlization. Is there any way. In hardware part..or anything..

Thnks in adv,
Sree
*** Let us share the knowledge ***
Jeff Schussele
Honored Contributor

Re: Very high CPU utlization

Hi,

I'd start with a thorough review of your kernel parameters. Particularly the timeslice parm.
IF it's set to less than 10 that could cause a lot of unnecessary context switching & needless overhead. Set it to 10 if it's not already there. Also check the dbc_max/min_pct values.
No need to have the OS cache the Informix reads/writes if the DB is already doing it.
Handling a large buffer cache can put quite the load on a system. We've found that between 400 & 800 MB tends to be a sweet spot for buffer cache size. There are other parms that can affect system perf - so a thorough review is required here.

Rgds,
Jeff
PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
Sridhar Bhaskarla
Honored Contributor

Re: Very high CPU utlization

Hi Sreekanth,

You would use psets if you want to bind processes to only certain number of CPUs. It may be required when you have few processes on the system that are monopolizing the CPUS.

It's very good if the CPU is running at 90-95% and *really* bad if they are running constantly at 100%. You would know why if you support production systems. It's just a matter of 5%-10% CPU and that allows you to plan for future. I may save a little money if I keep my systems at 100% but lose a lot if my production is affected.

I am not sure if it is because of pset configuration. PSETs are useful if the process groups are independent. For ex., you have two applications and they don't interact with each other. The performance will degrade if you have processes that are dependent on the processes in a different pset.

So, try without psets and see how the system responds. Also, look at HP's itrc site to see if there are any patches for PSETs.

Last but not the least, code is the major issue in most of the issues. It can make a lot of difference.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
Steve Lewis
Honored Contributor

Re: Very high CPU utlization

Informix is heavy on the following calls:
select()
read()
write()
semctl()

The more i/o you do, to the disk or the network, the more read/write calls you will do.
If you can offload some of the database connections from tcp socket onto shared memory/ipc you will reduce the calls.

If you are using KAIO, your system time will increase. Without KAIO and with AIOVPs, your system % reduces, but user % increases.

Having cpus at 99% is commonly a symptom of not using database indexes efficiently, doing sequential scans. Have you reviewed the indexes and updated the statistics recently?


Sreekanth Makam
Occasional Advisor

Re: Very high CPU utlization

Hi Steve Lewis,

*** Let us share the knowledge ***
Sreekanth Makam
Occasional Advisor

Re: Very high CPU utlization

Hi Steve Lewis,

I created indexes, done update stat..
All my queries are taking only index paths only.
I done database fragmentaion also. I distributed across 14 dbspaces. My whole databse is of 10GB. After this only actuall i got good reslts...

Any way..Thnks to all of them who participated in this...

Iam working in perfomance tuning on unix from last 1 year. So i will start new threads like this every day..Please participate in them. I am planing for this..Every day one topic will be there..Show your inteligence..And i will give realtime results for your thinkings...


***Let us share the knowledge****


Bye,
Sree
*** Let us share the knowledge ***
Ted Buis
Honored Contributor

Re: Very high CPU utlization

There is a whole book entitled HP-UX 11i Performance and Tuning, by Robert F. Sauers. You can get it from Amazon. It will help you with methodology as well as tools. I would recommend you consider it.
The HP-UX Performance Cookbook has also been posted to this forum before but I guess I will attached again.
Mom 6
Bill Hassell
Honored Contributor

Re: Very high CPU utlization

Your code is likely causing the high system load. If there was a magic parameter that reduced the load, everyone would use it. But if your program is making a million calls to poll(2) every second, there's nothing you can do the kernel to reduce all the steps in the poll code...it has to execute and will consume time. I use poll as an example since there are two ways to determine if mailbox I/O is ready: poll as fast as you can (massive system overhead) or setup a wait and let the program get signaled when the I/O ready. Just an example...


Bill Hassell, sysadmin
Tim D Fulford
Honored Contributor

Re: Very high CPU utlization

Hi

A couple of things for you to look at

1 - what is the read:write ratio you should be reading more than writing. If not why are you writing data ypu will not read!!
2 - You say that you are using indexes everywhere. I hope you do not mean this 100%. For instance "select * from table where tag_id not like 100" is MOST efficient when sequentially scanning the table and avoiding the index.
3 - What are the service times on the disks? If the disks are slow then informix/application may be waiting on the slow disk. striping etc may help (depend on result)
4 - when you do "onstat -g glo" the cpu-vp's should be using the most resource, then aio-vps, everything else should be way below these. Make sure your shm-vp & soc-vp (if they exist) are NOT heavy uses of resource. If they are, then move the poll threads to the cpu-vps. From what you said you should have 8-9 cpu-vps & i'd use 8-9 poll threads of the appropriate protocol.
5 - lastly.. and probably should have been top of the list. Are you getting the workload out you hoped for? If so then Bils comment is on-the-button. Though I personally do not like to see cpu's above 80% on a regular basis as I like to have something in my back pocket!! (now I get the sales speach on iCOD.. I do know about this, just have not used it)

Regards

Tim
-
Sreekanth Makam
Occasional Advisor

Re: Very high CPU utlization

hi tim,

1.The disk service time is less as iam fragmenting the tables across 12 db spaces which lie on diffrent disks. And keeping indexes across 4 db spaces. As aim using EVA SAN it will do striping by default.

If i watch dbspace utilization on onperf tool, all db spaces are utilizing uniformly.
Can you say what is the maximum number they can go. I mean no of read / wrties per second. By this i can conferm that disk is ok.

2. I checked onstat -g glo.. in that i observed

soc 41.19 461.68 502.87
soc 41.57 220.52 262.09
soc 42.56 230.36 272.92
soc 42.22 225.04 267.26
soc 43.88 236.15 280.03
soc 41.65 223.41 265.06

Can u coment anything on this...

3. Up to know glance plus was showing network bottelneck. Now after putting seperate test lan. It is solved.

***Problem is problem until u feel that is not a problem****




*** Let us share the knowledge ***
Tim D Fulford
Honored Contributor

Re: Very high CPU utlization

Hi

skipping to the onstat -g glo..

you have 6 soc-vps evenly loaded(ish) What are the cpu-vps load in comparison? This is the key to tuning BALANCING the load. The best thing to post is the whole "onstat -g glo", but if you feel this is too much just the top bit which summarises the spead of the workload... to me your system should probably be optimised for local access (shared memory) and I'm guessing. If this is the case why do you have 6 soc-vps. My guess is that you may effectively have erronously assigned the soc-vps to the network vps (soc)
NETTYPE ipcshm,9,100,CPU
NETTYPE soctcp,6,100,CPU
NUMCPUVPS 9
This will assign the first 9 cpu-vps with shm-poll threads (sm_poll). The 6 socket poll threads WILL NOT go onto the cpu-vps so will end up as soc-vps (as yopu have). If this is you will need to decide what the split of shm connections & soc connections. say 2/3 1/3 split I would do the follwoing
NETTYPE ipcshm,6,100,CPU
NETTYPE soctcp,3,100,CPU
NUMCPUVPS 9
now all the poll threads will get onto each of the CPU-vps.

so to summarise
1 - onstat -g glo (the whole thing please)
2 - onstat -c or ONCONFIG would also be useful..

Timmmmy
-