Linux system level CPU utilization spikes

[Eric O'Brien]

I am seeing occasional system level CPU spikes and am unable to zero in on a cause. Throwing this out there to see if anyone can point me in the right direction. Besides this kernel cpu spike, everything else seems fairly quite. Any words of wisdom would be appreciated.
Thanks

I have pretty much just one Java tomcat application running on this system with the following opts
JAVA_OPTS="-server -Xmx1280m -Xms1280m"

==> uname -rv
2.6.11.4-20a-bigsmp #1 SMP Wed Mar 23 21:52:37 UTC 2005

Here's a vmstat that catches my issue:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 0 204284 140860 2875292 0 0 0 0 1030 209 26 0 74 0
1 0 0 203628 140860 2875292 0 0 0 176 1177 462 26 1 72 0
1 0 0 203756 140860 2875292 0 0 0 224 1149 288 26 0 74 0
1 0 0 203564 140860 2875292 0 0 0 0 1205 358 26 0 74 0
88 0 0 203564 140860 2875292 0 0 0 240 1747 330 17 44 38 0
90 0 0 203564 140860 2875292 0 0 0 0 1374 169 9 91 0 0
92 0 0 203596 140860 2875292 0 0 0 12 1315 177 8 92 0 0
93 0 0 203596 140860 2875292 0 0 0 0 1243 160 9 91 0 0
93 0 0 203692 140860 2875292 0 0 0 0 1279 153 8 92 0 0
93 0 0 203788 140860 2875292 0 0 0 0 1018 113 9 91 0 0
94 0 0 203788 140860 2875292 0 0 0 8 1024 101 9 91 0 0
95 0 0 203868 140860 2875292 0 0 0 0 1028 101 9 91 0 0
96 0 0 203868 140860 2875292 0 0 0 0 1017 102 9 91 0 0
96 0 0 203868 140860 2875292 0 0 0 0 1024 99 8 92 0 0
97 0 0 203868 140860 2875292 0 0 0 0 1021 103 9 91 0 0
1 0 0 203732 140860 2875292 0 0 0 0 1066 316 12 74 14 0
3 0 0 203764 140860 2875292 0 0 0 0 1199 520 23 2 75 0
1 0 0 203948 140860 2875292 0 0 0 0 1159 342 24 0 75 0
1 0 0 203756 140860 2875292 0 0 0 346 1136 317 26 0 74 0


Here's some other ancillary info from the system:

==> cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 1.80GHz
stepping : 7
cpu MHz : 1794.467
cache size : 512 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 3547.13

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 1.80GHz
stepping : 7
cpu MHz : 1794.467
cache size : 512 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 3579.90

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 1.80GHz
stepping : 7
cpu MHz : 1794.467
cache size : 512 KB
physical id : 3
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 3579.90

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 1.80GHz
stepping : 7
cpu MHz : 1794.467
cache size : 512 KB
physical id : 3
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 3579.90

==> cat /proc/meminfo
MemTotal: 3827500 kB
MemFree: 198804 kB
Buffers: 140872 kB
Cached: 2878364 kB
SwapCached: 0 kB
Active: 1826600 kB
Inactive: 1650012 kB
HighTotal: 2948992 kB
HighFree: 41856 kB
LowTotal: 878508 kB
LowFree: 156948 kB
SwapTotal: 1052248 kB
SwapFree: 1052248 kB
Dirty: 300 kB
Writeback: 0 kB
Mapped: 481840 kB
Slab: 134864 kB
CommitLimit: 2965996 kB
Committed_AS: 1638156 kB
PageTables: 2056 kB
VmallocTotal: 112632 kB
VmallocUsed: 11496 kB
VmallocChunk: 100860 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

[Stuart Browne]

Well you see the sudden increase in 'runnable' processes, you need to find out what process it was that suddenly launched/woke up almost 100 processes at the same time.

Look at the usual suspects like cron or java garbage collection.

[Eric O'Brien]

No cron jobs, double checked that. I was thinking java gc, but I have partial and full garbage collection events every second when the system is fine.
I also wasn't sure if the gc events would show up in the User or system bucket.
I'll try catching the gc events during spikes.

[skt_skt]

"run queue" is above 90 processes that time..and at the same time the "blocked queue" is still zero.. So CPU are working for those 90 processes...you have to find what are these processes and where are they coming from

[Eric O'Brien]

Digging into Java GC a little more turned up the issue. I thought Full GC's every second weren't good, but I had plenty of available heap. Using "jstat -gccause" to drill into why the GC's were happening turned up "System.gc()" being called a ton. I can't modify the code, but I modified my java options to disable the explicit GC:

-XX:+DisableExplicitGC

And poof... GC is almost nill and user CPU utilization has dropped significantly and I haven't seen the system CPU spike yet.
Thanks for the responses