performance issue

[K.C. Chan]

All,
I have a server with 2 cpu at 2.4Ghz. It is being stress beyond belief. Here's the output of vmstat:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
22 0 2 133480 12600 196336 2250868 0 0 0 161 133 600 96 4 0
28 0 2 133480 12904 196352 2251416 0 0 0 222 145 617 96 4 0
21 0 2 133480 13260 196384 2253220 0 0 0 1112 259 542 79 21 0
19 0 2 133480 13084 196404 2256200 0 0 0 684 201 556 87 13 0
20 0 1 133480 11848 196412 2258740 0 0 0 1032 247 553 80 20 0
19 0 2 133480 11132 196420 2259032 0 0 0 121 132 618 97 3 0
19 0 2 133480 11780 196444 2253352 0 0 0 142 133 652 95 5 0
17 0 1 133480 13936 196452 2253676 0 0 0 163 134 635 86 14 0
As you can see, some of the processes are swapping out and there are way too much processes waiting to be service. Is there a way to get the powers to be to acknowledge that the server is overloaded? I am thinking of graphing it? Does any one know of an easy tool which could graph this sort of data(larrd, rddtool, mrt...?).
On another note, I noticed there are lots of context switching occuring; How does this have an impact on cpu load? I am under the impression that the more occurence of context switch taking place, the more load is being put on the cpu. Is there a way to relate this (show the big cheese) the relationship btw context switch and load on cpu? If someone knows of a good doc which explains some performance metrics on linux, then could you share it? Thanks.

[Murat SULUHAN]

Hi

Do you send us more information about, how many megabytes of ram, swap size, distro, kernel version, running services and also ps axu output

I think this url may be a good starting documentation
http://linuxperf.nl.linux.org/

Best Regards
Murat SULUHAN

[Steven E. Protter]

You have vmstat on Linux?

I didn't know it was there.

I'm attaching a set of performance scripts that are mostly sar. You will have to adjust them, but you can collect data on Linux with them.

This is an HP-UX script.


SEP

[K.C. Chan]

profile of server
redhat 7.2
kernel 2.4.18
memmory 3732948Kbytes
swap 2097136Kbytes

[K.C. Chan]

attached is the output of ps axu

[Mark Grant]

Sorry but to me, this machine isn't swapping. It has swapped at some point but isn't in the output of vmstat. The number of context switches is not excessive by any means though I would worry about the CPU idle time.

My guess would be that whatever /usr/local/myap is is causing a problem. It seems to use a fair bit of RAM and some of them, at least, seem to be putting a load on the CPU.

Where you might see some disk throughput problems is that journald and your raid daemon appear to have been swapped out at some point.

You might have an argument for increasing the power of the machine but to me it looks like there is a stronger argument for doing an audit of /usr/local/myap.

[K.C. Chan]

Let me clearify what I meant by swapped out, I mean some process are showing up "w" column; according to the man on vmstat the process are swapped out but still runnable.

I know the cause are the app which you have pointed out; unfortunately I could only predict and gave my two cent regarding how this is implemented. What makes this worst is that the processes are real time process and there are lots of them. When you say audit, do you mean the account utility under Linux? BTW, where do you see that journald and raid daemon being swapped out, how can I tell? Thanks.

[Mark Grant]

K.C.Chan,

THe output of your ps shows commands surrounded with square brackets. This indicates that those processes are swapped but runnable.