Performance Bottleneck

[RAKESH_23]

I suspect that my system has a CPU bottleneck..
I did following things..

1) Run the glance...Memory is OK at 56% and swap is utilized at 33% ...also vmstat no paging( pi,po,sr in vmstat ) so ruled out memory problems.
2) Disk IO : wait I/O is 1% in sar and also vmstat no wait stats reported
3) Avg CPU uti is 65%.. User mode is 10% and 55% is system mode. in vmstat, averge run queue is 5..so if run queue is high why cpu is only 65% busy and specially in system mode...and also in glance vfaults is 45% on average..is it bad...

[Sajith V Mannadiar]

Rakesh,

55% in system mode is not good. what is the Context switch util?

Is there any runaway processes?

Try, ps -elf|grep 'R '

[Steven E. Protter · Accepted solution]

This could have something to do with how the process threads are handled in the application.

Note swap utilized at 33% really means 33% of it is reserved and no paging is going on.

You could have processes running in real time mode and sleeping, preventing the processor from doing anything else.

I'd collect some more comprehensive data (See attachment):

I wonder if you have recently changed your time slice or other kernel parameters.

A kernel review seems a good first step. Please see this doc:
http://www2.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-1335382922+1102697480822+28353475&searchCrit=allwords

SEP

[RAKESH_23]

Thanks for the reply..

here attaching vmstat output.

1) ps -efl |grep R will give any line containing letter "R" ..

Steve,
I could not open the link that u sent..will adding the CPU help me ?

any other help is appreciated..

[RAKESH_23]

Some more info..

1) Using glance i saw the system call for java process .rate of sigcleanup is very very high it is 47320
2) most of the threads for this process are in wait reason:SYSTEM

any clue, what sigcleanup does and why it can be so high..
thanks

[Tim D Fulford]

Hi

The reason why you are blocked on SYSTEM & the system % CPU is high are probably the same.

I *suspect* you have a runaway process. I saw this only yesterday (29 Jan) when a script was continually spawing and doing "ls -ltr". There was only 10-20 at any one time but is was happening so continually that the system CPU usage was 100% (20% usr 80% sys). I'm guessing you have something similar, but probably not "ll -ltr".

The way I spotted it was to look at
o ps -ef
o top
o glance.
It became apparent soon that this was the case. Once I figured out what it was I was able to trace it back using the following scripts looking for the ppid & pppid & ppppid etc...

for ppid in $(ps -el | perl -ane 'if m/\bll\b/) {print "$F[4]\n"}')
do
ps -fp $ppid
done

& the next layer

for ppid in $(ps -el | perl -ane 'if (m/\bll\b/) {print "$F[4]\n"}')
do
for pppid in $(ps -lp $ppid | egrep $ppid | perl -ane 'print "$F[4]\n"')
do
ps -fp $pppid
done
done

& the next layer...

for ppid in $(ps -el | perl -ane 'if (m/\bll\b/) {print "$F[4]\n"}')
do
for pppid in $(ps -lp $ppid | egrep $ppid | perl -ane 'print "$F[4]\n"')
do
for ppppid in $(ps -lp $pppid | egrep $pppid | perl -ane 'print "$F[4]\n"')
do
ps -fp $ppppid
done
done
done

etc...

Regards

Tim

[Tim D Fulford]

just one other thing. The attachment looks like a vmstat out put rather than ps -fe | egrep R... could you post the ps -fe | egrep R ?

Tim