topic Re: Possible Server Performance Issue in Operating System - Linux

Possible Server Performance Issue

Steve Burt_1 — Mon, 12 Feb 2007 09:27:39 GMT

I say possible, because I dont know whether or not they are related

Facts
Red Hat Enterprise Linux AS release 4 (Nahant Update 1) 2.6.9-11.ELsmp

Symptom 1

dmesg reports

Losing some ticks... checking if CPU frequency changed.

Sympton 2

High CPU usage yet no apparent process seems to report the consumption, this is a weblogic server, interesting enough when you stop weblogic then the CPU load drops to minimal usage...

Yes you might say well its weblogic, but something in me says otherwise, but I have no logical facts... am i looking into this too much..

Appreciate any kind of help...

Though the server is experiencing CPU loads of 90 - 100% there is still plenty of memory available and swap is not even being touched..

This is a DL580 with 2 Intel(R) Xeon(TM) MP CPU 3.00GHz.. can anyone help me in resolving this ...

Much appreciated

Re: Possible Server Performance Issue

Wouter Jagers — Mon, 12 Feb 2007 10:16:09 GMT

Strange.. have you checked the CPU settings in your BIOS as well ?

And what does "cat /proc/cpuinfo" say ?

Re: Possible Server Performance Issue

Steve Burt_1 — Mon, 12 Feb 2007 10:21:35 GMT

Hi Wouter,

here is the cpuinfo

processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) MP CPU 3.00GHz
stepping : 1
cpu MHz : 3001.341
cache size : 1024 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_c
pl est tm2 cid xtpr
bogomips : 5881.85

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) MP CPU 3.00GHz
stepping : 1
cpu MHz : 3001.341
cache size : 1024 KB
physical id : 3
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_c
pl est tm2 cid xtpr
bogomips : 5980.16

Re: Possible Server Performance Issue

Steve Burt_1 — Mon, 12 Feb 2007 10:22:58 GMT

Sorry bit trigger happy on the submit button, I havent had chance to check the CPU settings, but I know that Hyperthreading is enabled..

--Steve

Re: Possible Server Performance Issue

Wouter Jagers — Mon, 12 Feb 2007 13:38:40 GMT

Are you using Intel SpeedStep ? I read that might cause such issues.. If so, try disabling it.

Re: Possible Server Performance Issue

Steve Burt_1 — Mon, 12 Feb 2007 14:36:10 GMT

Looks like I will have to bring the server down, and go into the bios, as there certainly appears to be no processes running... Actually looking at the specs for a DL580 there is indeed an option to turn it off, this sounds like a quick win, I have found some other articles that suggest passing noapic to the kernel...

Will keep you posted...

Re: Possible Server Performance Issue

Steve Burt_1 — Mon, 12 Feb 2007 17:40:58 GMT

Further research from the following page http://softwarecommunity.intel.com/ISN/Community/en-US/forums/thread/30227610.aspx

Tells me that the Enhanced Intel Speedstep technology is supported, notice the est in the flags section, in /proc/cpuifo, but as you quite rightly said who's to say that it indeed works...

Re: Possible Server Performance Issue

rick jones — Tue, 13 Feb 2007 12:47:45 GMT

WRT the high CPU utilization, what does it say for the user/system (aka kernel) split in top? The CPU consumed by "networking" may not be charged to the correct process or sometimes even to a process.

Re: Possible Server Performance Issue

Steve Burt_1 — Wed, 14 Feb 2007 07:01:04 GMT

Hi Rick,

Here is my top listing...

top - 11:59:35 up 4:59, 5 users, load average: 7.45, 7.35, 6.67
Tasks: 105 total, 2 running, 103 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.0% us, 0.4% sy, 0.0% ni, 0.4% id, 0.0% wa, 0.0% hi, 0.2% si
Mem: 4086548k total, 1682556k used, 2403992k free, 66844k buffers
Swap: 2031608k total, 0k used, 2031608k free, 444836k cached

Re: Possible Server Performance Issue

rick jones — Wed, 14 Feb 2007 16:11:52 GMT

99% user - not to be pedandtic, but are you _sure_ no process is showing CPU utilization?

To see if it is an instrumentation error, you may need to try some sort of soaker process and see how fast it runs after a boot, and then how fast it runs when the system is in this presented state. If the soaker runs as well as before, it suggests an instrumentation issue.

Perhaps a system-level profiler - I am more familiar with what one can run on Integrity boxes, but perhaps oprofile can do it and show where time is being spent as a sanity check.

Re: Possible Server Performance Issue

Steve Burt_1 — Thu, 15 Feb 2007 03:52:41 GMT

Hi Rick, I know its strange but sadly true....

Tasks: 94 total, 2 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 85.1% us, 0.5% sy, 0.0% ni, 14.1% id, 0.1% wa, 0.0% hi, 0.2% si
Mem: 4086548k total, 2618700k used, 1467848k free, 134508k buffers
Swap: 2031608k total, 0k used, 2031608k free, 1098932k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9453 admin 15 0 2916 884 692 R 2 0.0 0:00.01 top
1 root 16 0 2892 552 472 S 0 0.0 0:01.22 init
2 root RT 0 0 0 0 S 0 0.0 0:01.48 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.27 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:01.53 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.28 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3
10 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/0
11 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/1
12 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/2
13 root 5 -10 0 0 0 S 0 0.0 0:00.00 events/3
14 root 6 -10 0 0 0 S 0 0.0 0:00.49 khelper
15 root 15 -10 0 0 0 S 0 0.0 0:00.00 kacpid
47 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0
48 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1
49 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/2
50 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/3
60 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
61 root 15 0 0 0 0 S 0 0.0 0:00.25 pdflush
63 root 7 -10 0 0 0 S 0 0.0 0:00.00 aio/0
64 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1
65 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/2
66 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/3
51 root 15 0 0 0 0 S 0 0.0 0:00.00 khubd
62 root 25 0 0 0 0 S 0 0.0 0:00.00 kswapd0
139 root 25 0 0 0 0 S 0 0.0 0:00.00 kseriod
217 root 5 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/0
218 root 7 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/1
219 root 5 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/2
220 root 11 -10 0 0 0 S 0 0.0 0:00.00 kmirrord/3
228 root 15 0 0 0 0 S 0 0.0 0:05.49 kjournald
1067 root 6 -10 2832 464 380 S 0 0.0 0:00.00 udevd
1488 root 19 0 0 0 0 S 0 0.0 0:00.00 kjournald
1644 root 17 0 4248 1360 1124 S 0 0.0 0:00.06 rc
1720 root 16 0 2536 456 396 S 0 0.0 0:13.60 cpuspeed
2171 root 16 0 1704 644 552 S 0 0.0 0:00.16 syslogd
2175 root 15 0 3064 484 420 S 0 0.0 0:00.01 klogd
2185 root 16 0 2980 484 412 S 0 0.0 0:02.19 irqbalance
2229 root 16 0 5912 1052 876 S 0 0.0 0:00.35 rpc.idmapd
2268 rpc 16 0 1924 604 512 S 0 0.0 0:00.00 portmap
2325 ntp 16 0 6324 6324 3436 S 0 0.2 0:02.45 ntpd
2371 root 18 0 1616 552 480 S 0 0.0 0:00.00 acpid
2664 root 16 0 112m 1100 776 S 0 0.0 0:02.10 hpasmd
2694 root 16 0 12572 1332 684 S 0 0.0 0:00.16 cmapeerd
2695 root 16 0 2308 1392 744 S 0 0.0 0:00.73 cmathreshd
2696 root 16 0 12504 1244 864 S 0 0.0 0:35.92 cmahostd
2736 root 16 0 22280 1032 772 S 0 0.0 0:00.35 cmastdeqd
2743 root 15 0 64332 924 692 S 0 0.0 0:05.00 cmahealthd
2745 root 16 0 4016 1236 620 S 0 0.0 0:27.49 cmaperfd
3042 root 15 0 23760 6324 2632 S 0 0.2 0:13.36 snmpd
3051 root 16 0 8804 2124 1692 S 0 0.1 0:00.05 cupsd
3085 root 16 0 4204 1668 1376 S 0 0.0 0:00.00 sshd
3098 root 16 0 3192 860 728 S 0 0.0 0:00.00 xinetd
3116 root 16 0 7880 3124 2084 S 0 0.1 0:01.16 sendmail
3124 smmsp 16 0 6876 2616 1796 S 0 0.1 0:00.00 sendmail
3134 root 16 0 2772 540 468 S 0 0.0 0:00.06 gpm
3143 root 16 0 4708 1116 724 S 0 0.0 0:00.04 crond
3174 xfs 16 0 3932 1688 864 S 0 0.0 0:00.03 xfs
3191 root 16 0 3680 740 624 S 0 0.0 0:00.00 atd
3200 dbus 16 0 3592 1236 1092 S 0 0.0 0:00.01 dbus-daemon-1
3210 root 16 0 4732 552 492 S 0 0.0 0:00.06 rhnsd
3219 root 16 0 7796 5216 1600 S 0 0.1 0:23.22 hald

Re: Possible Server Performance Issue

rick jones — Thu, 15 Feb 2007 12:31:30 GMT

The Phantom Process strikes again!-) I think the profile route is the next thing to do to see what if anything is actually consuming CPU cycles. Alas, I know Caliper and q-syscollect, but both of those are Itanium - oprofile exists for x86 but I am unable to suggest command syntax.

Could also compare ps -ef output to the process listing of top I suppose.

Re: Possible Server Performance Issue

Steve Burt_1 — Tue, 20 Mar 2007 06:58:30 GMT

The Phantom process has been discoverd.

Interogating Oracle produced a piece of memory intensive SQL running 5000 a second..

Apologies for delay sickeness and paternity leave delayed a responce

Thanks for all your help...