Possible Server Performance Issue

[Steve Burt_1]

I say possible, because I dont know whether or not they are related

Facts
Red Hat Enterprise Linux AS release 4 (Nahant Update 1) 2.6.9-11.ELsmp


Symptom 1

dmesg reports

Losing some ticks... checking if CPU frequency changed.

Sympton 2

High CPU usage yet no apparent process seems to report the consumption, this is a weblogic server, interesting enough when you stop weblogic then the CPU load drops to minimal usage...

Yes you might say well its weblogic, but something in me says otherwise, but I have no logical facts... am i looking into this too much..

Appreciate any kind of help...

Though the server is experiencing CPU loads of 90 - 100% there is still plenty of memory available and swap is not even being touched..

This is a DL580 with 2 Intel(R) Xeon(TM) MP CPU 3.00GHz.. can anyone help me in resolving this ...

Much appreciated

[Wouter Jagers]

Strange.. have you checked the CPU settings in your BIOS as well ?

And what does "cat /proc/cpuinfo" say ?

[Steve Burt_1]

Hi Wouter,

here is the cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) MP CPU 3.00GHz
stepping : 1
cpu MHz : 3001.341
cache size : 1024 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_c
pl est tm2 cid xtpr
bogomips : 5881.85

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) MP CPU 3.00GHz
stepping : 1
cpu MHz : 3001.341
cache size : 1024 KB
physical id : 3
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_c
pl est tm2 cid xtpr
bogomips : 5980.16

[Steve Burt_1]

Sorry bit trigger happy on the submit button, I havent had chance to check the CPU settings, but I know that Hyperthreading is enabled..

--Steve

[Wouter Jagers]

Are you using Intel SpeedStep ? I read that might cause such issues.. If so, try disabling it.

[Steve Burt_1]

Looks like I will have to bring the server down, and go into the bios, as there certainly appears to be no processes running... Actually looking at the specs for a DL580 there is indeed an option to turn it off, this sounds like a quick win, I have found some other articles that suggest passing noapic to the kernel...

Will keep you posted...

[Steve Burt_1]

Further research from the following page http://softwarecommunity.intel.com/ISN/Community/en-US/forums/thread/30227610.aspx

Tells me that the Enhanced Intel Speedstep technology is supported, notice the est in the flags section, in /proc/cpuifo, but as you quite rightly said who's to say that it indeed works...

[rick jones]

WRT the high CPU utilization, what does it say for the user/system (aka kernel) split in top? The CPU consumed by "networking" may not be charged to the correct process or sometimes even to a process.

[Steve Burt_1]

Hi Rick,

Here is my top listing...


top - 11:59:35 up 4:59, 5 users, load average: 7.45, 7.35, 6.67
Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.0% us, 0.4% sy, 0.0% ni, 0.4% id, 0.0% wa, 0.0% hi, 0.2% si
Mem: 4086548k total, 1682556k used, 2403992k free, 66844k buffers
Swap: 2031608k total, 0k used, 2031608k free, 444836k cached

[rick jones]

99% user - not to be pedandtic, but are you _sure_ no process is showing CPU utilization?

To see if it is an instrumentation error, you may need to try some sort of soaker process and see how fast it runs after a boot, and then how fast it runs when the system is in this presented state. If the soaker runs as well as before, it suggests an instrumentation issue.

Perhaps a system-level profiler - I am more familiar with what one can run on Integrity boxes, but perhaps oprofile can do it and show where time is being spent as a sanity check.