server 100% in system mode

[Sajeesh O.K]

Hi

I have a linux box with RHAS 2.1.
The configuration is 4x1.5 GHz CPU and
4 GB RAM.

My top output showing server is 100% in system mode and load also high. Please see the output below.

273 processes: 261 sleeping, 12 running, 0 zombie, 0 stopped
CPU0 states: 100.0% user, 0.0% system, 100.0% nice, 0.0% idle
CPU1 states: 100.0% user, 0.0% system, 100.0% nice, 0.0% idle
CPU2 states: 100.0% user, 0.0% system, 100.1% nice, 0.0% idle
CPU3 states: 92.0% user, 7.0% system, 88.0% nice, 0.0% idle
Mem: 3831076K av, 1600676K used, 2230400K free, 0K shrd, 215044K buff
Swap: 2044056K av, 0K used, 2044056K free 493872K cached

Also my kernel parameter setting below.
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_forward = 0
kernel.sysrq = 0
kernel.msgmax = 16384
kernel.msgmnb = 32768
kernel.msgmni = 2052
kernel.shmmax = 2147483648
fs.file-max = 417145
kernel.sem=250 32000 128 128
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time=300
net.core.rmem_default = 262143
net.core.rmem_max = 262143
net.core.wmem_default=262143
net.core.wmem_max = 262143
net.ipv4.tcp_rmem = 4096 262143 262143
net.ipv4.tcp_wmem = 4096 262143 262143
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
vm.freepages = 1024 2048 3072
net.ipv4.tcp_max_syn_backlog=8192

Can somebody help what could be the problem.

Thanks
Sajeesh

[Sajeesh O.K]

Opps ! It is 100% in nice mode !

[Vitaly Karasik_1]

do you have manu user's tasks? ["top" output]

[Fred Ruffet]

what are sthose 12 process runing ? What are they doing ?

Regards,

Fred

[HGN]

Hi

What is the application running on the system & what are the top process running , was it always doing since the application was loaded or recently have you made any changes by installing something or modifying the kernel parameters which might also affect the performace or on top of everything there might be a real resouce crunch wherein you might have to consider adding more resource all can be told with more details.

Rgds

HGN

[Johannes Krackowizer_1]

hi sajeesh,

i have see some similar problems under 2.0 and 2.1 kernels. you should not worry about this if there is a huge nice level. i think you have cpus where the kernel should do nice calls so the cpu's know that there is nothing to do, but the kerne don't make this calls. so the only thing is your system needs more power than needed. if your system runs fine hope that a newer kernel will fix this bug.

best regards,

johannes

[Mike Jagdis]

You're running HP daemons, right? You probably have several disks too...

I just posted this in the "cmascsid process taking up all of CPU" thread. I bet it's the same...

-------------------------------------------

The explanation is somewhat involved...

If you trace cma*d you'll find that it doesn't do anything but open the device, ioctl, close. Admittedly rather more times than should be necessary but that's just incidental bad design.

You'll find the delay - and system time consumption - seems to happen on the close. From here you need a fairly good working knowledge of the Linux kernel...

Ok? still with me then?

Run oprofile for a while and you'll find the cpu time is being consumed by invalidate_bdev. Which is interesting :-).

Invalidate_bdev is called from kill_bdev. Kill_bdev is called from the block device release code. Release is what happens on last close. Now the monitoring daemon is opening the unpartitioned disk device which it is pretty certain nothing else has open. (Off hand I'm not sure if even having an fs on the device counts as it being open. There are subtle differences and I *think* I'm right in saying that block device access and fs access is considered different at this level. Don't quote me or blame me!)

So, each close triggers invalidate_bdev. Why is this so bad? Well, the idea is that when the last close happens on a device you need to flush any cached data because, with much PC HW, you can't be sure when the media gets changed. Invalidate_bdev isn't *meant* to be called often. It works by scanning through the entire list of cached data for block devices to find and drop data related to the device being closed. So it sucks system time and the amount is proportional to the amount of cached (from any device) data you have.

WORKAROUND:
All you need to do is to make sure that each time the cma*d daemon closes the device it isn't the *last* close - i.e. some other process has the device open. The other process doesn't even need to *do* anything. Try something along the lines of:

sh -c 'kill -STOP $$' < /dev/cciss/c0d0 > /dev/null 2>&1 &

Hope that's all clear! (As mud... :-) )

(HP: As well as blind debugging I do Linux & OSS consultancy. I happen to know the answer to this one as it came up at a major investment bank...)