Top Output.

[Ian Green]

Hi There:

I am running a load test against an application and the application seems to cpu intensive. I find the following...
we have 6 CPU's and the IDLE portion on all of them are 0. The avg. USER portion is 35 and the SYS portion is 65. This was for about 120 users logged on the system.There was no memory issue here.
My Question:

1. I know i have to get back to the application to rewrite the code but what are the reports i need to give them which may help them identify the root cause.

This is what i currently have..

1.output of lsof( ihave like lsof|grep username for the top user on top). Is it OK?
2.i did find netstat -a |grep wait and found in the last two coloumns

locald1.8080 TIME_WAIT
mem370.7710 TIME_WAIT
There were 181 connections this way.Is it anything wrong. Actually i think the application is accessing some backend server(mem370) but what is locald1.8080?

Please help me to find the root cause.

Thanks
Ian.

[Sandip Ghosh]

It looks like the Kernel parameters of the system is not set properly. Because system is taking around 60 to 65%. If the kernel parameters are set properly then the system portion should not exceed more than 20 to 25%. Look at the glance and find out the condition of the file tables parameter.

Can you post your kernel parameters, sar -v 2 2?

Snadip

[Ian Green]

Hi Sandip:

I have attached the sar -v output. Please have a look.

Thanks
Ian.

[Martin Johnson]

A company called precise (http://www.precise.com) has once of the best suite of products to monitor the performance of an application. Their products can tell you where you are spending most of your time. You can drill down an see the line of SQL code that is using the most resources.

Their products are not cheap, but if you have a mission critical app that absolutely has to be a top performer, the precise product suite is the way to go.


Marty

[James R. Ferguson]

Hi Ian:

The TIME_WAIT is a normal socket transition state.

As suggested, have a look at your system with 'glance'. If you need, there is a trial version on the Application CDROMs. If you don't license it, it is well worth the cost.

Look at the basic I/O, memory metrics and, particularly the CPU screens which summarize system overhead.

Make sure that your kernel timeslice parameter is set to <10>. If it is too low (as some DB templates set it) you will do nothing but context (stack) switch, which would cause you to expend lots of CPU "doing much of nothing".

Regards!

...JRF...

[Sandip Ghosh]

It seems the ninode of your system is set too high. Actually ninode is used for hfs filesystems only. Vxfs file system uses the dynamic inodes. You can bring down your ninode to 4000.

But for ninode your system should not take 60% of your system overhead. There must be something else. There must be some problem with some other kernel parameters. By the way, do you have any runaway processes?

Sandip

[Jeff Schussele]

Hi Ian,

High Sys usage could be indicative of bottle necks. If you have glance - run it up & if any of the the other (disk/mem/swap) are as high as CPU - then you definitely are experiencing bottlenecks.

If mem usage is high +85% & swap is very active - could be mem bottleneck.

Then run

sar -u 5 10

And check the WIO value if that's high then the CPU is waiting on resources to free up.
Then you can start digging into just exactly what resources it's waiting on.
And yes it certainly could be due to crappy code....

Rgds,
Jeff

[Ian Green]

Hi:

I do understand that the ninode value is high but that should not hurt. There are no runaway process but all the 6 cpu's are filled up (ist three > 65% and other 3 around 20% with the average SYS as 65. Also the timeslice is set to 10. This is 11.0 with 20 gig ram.6 cpu's.(750 mhz) nclass.

Thanks
Ian.

[Martin Johnson]

Use glance to monitor individual application process. Assuming this is an HPUX system, look at the process system calls. This will give you some idea where the time is being spent.

Marty