slabtop

[skt_skt]

i am getting following informations from slabtop
Red Hat Enterprise Linux AS release 3 (Taroon Update 9)
High load average

14:24:43 up 8 days, 12:28, 9 users, load average: 20.59, 24.67, 24.12

2.4.21-53.ELsmp

free -mt
total used free shared buffers cached
Mem: 8006 7966 40 0 84 827
-/+ buffers/cache: 7054 952
Swap: 8143 6789 1354
Total: 16150 14755 1394

# vmstat 3 3
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
17 0 6954480 43164 85932 847572 12 6 10 6 1 3 10 3 6 0
14 0 6954388 42948 86072 847576 29 0 29 276 230 4869 97 3 0 0
18 0 6954312 42860 86108 847580 25 0 25 186 187 6908 100 0 0 0




# head slabtop.txt
Active / Total Objects (% used) : 259866 / 494614 (52.5%)
Active / Total Slabs (% used) : 27459 / 27544 (99.7%)
Active / Total Caches (% used) : 50 / 76 (65.8%)
Active / Total Size (% used) : 67179.68K / 102304.61K (65.7%)
Minimum / Average / Maximum Object : 0.01K / 0.21K / 128.00K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
198310 84552 42% 0.11K 5666 37 22664K buffer_head
54326 27499 50% 0.16K 2362 24 9448K skbuff_head_cache
51570 25431 49% 0.25K 3438 16 13752K size-256

here is the behaviour i see wit the java processes/multiple which are taking higher CPU

]# strace -p 25798
Process 25798 attached - interrupt to quit
gettimeofday({1206445316, 628819}, NULL) = 0
futex(0x8112054, FUTEX_WAKE, 1) = 1
futex(0x8112050, FUTEX_WAKE, 1) = 1
futex(0x805c588, FUTEX_WAKE, 1) = 1
futex(0x805c488, FUTEX_WAIT, 2, NULL) = 0
futex(0x81aa44c, 0x4 /* FUTEX_??? */, 1) = 1
futex(0x805c488, FUTEX_WAKE, 1) = 1
futex(0x805c508, FUTEX_WAIT, 2, NULL) = 0
futex(0x805c508, FUTEX_WAKE, 1) = 0
futex(0x805c608, FUTEX_WAKE, 1) = 1
futex(0x6cebd548, FUTEX_WAKE, 1) = 1
futex(0x6cebd54c, FUTEX_WAIT, 491169, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x805c608, FUTEX_WAKE, 1) = 0
gettimeofday({1206445317, 864456}, NULL) = 0
futex(0x8112054, FUTEX_WAKE, 1) = 1
futex(0x8112050, FUTEX_WAKE, 1) = 1
futex(0x805c588, FUTEX_WAKE, 1) = 1
futex(0x805c488, FUTEX_WAIT, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x81aa44c, 0x4 /* FUTEX_??? */, 1) = 1
futex(0x805c488, FUTEX_WAKE, 1) = 1
futex(0x805c508, FUTEX_WAIT, 2, NULL) = 0
futex(0x805c508, FUTEX_WAKE, 1) = 0
futex(0x805c608, FUTEX_WAKE, 1) = 1
futex(0x6cebd548, FUTEX_WAKE, 1) = 1
futex(0x6cebd54c, FUTEX_WAIT, 491171, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x805c608, FUTEX_WAKE, 1) = 0
gettimeofday({1206445319, 214760}, NULL) = 0
futex(0x8112054, FUTEX_WAKE, 1) = 1
futex(0x8112050, FUTEX_WAKE, 1) = 1
futex(0x805c588, FUTEX_WAKE, 1) = 1
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0xb7d70e50) = -1216655360
futex(0x805c488, FUTEX_WAIT, 2, NULL) = 0
futex(0x81aa44c, 0x4 /* FUTEX_??? */, 1) = 1
futex(0x805c488, FUTEX_WAKE, 1) = 1
futex(0x805c508, FUTEX_WAIT, 2, NULL) = 0
futex(0x805c508, FUTEX_WAKE, 1) = 0
futex(0x805c608, FUTEX_WAKE, 1) = 1
futex(0x6cebd548, FUTEX_WAKE, 1) = 1
futex(0x6cebd54c, FUTEX_WAIT, 491173, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x805c608, FUTEX_WAKE, 1) = 0
gettimeofday({1206445320, 590187}, NULL) = 0
futex(0x8112054, FUTEX_WAKE, 1) = 1
futex(0x8112050, FUTEX_WAKE, 1) = 1
futex(0x805c588, FUTEX_WAKE, 1) = 1
futex(0x81aa44c, 0x4 /* FUTEX_??? */, 1) = 1
futex(0x805c488, FUTEX_WAKE, 1) = 1
futex(0x805c508, FUTEX_WAIT, 2, NULL) = 0
futex(0x805c508, FUTEX_WAKE, 1) = 0
futex(0x805c608, FUTEX_WAKE, 1) = 1
futex(0x6cebd548, FUTEX_WAKE, 1) = 1
futex(0x6cebd54c, FUTEX_WAIT, 491175, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x805c608, FUTEX_WAKE, 1) = 0
gettimeofday({1206445322, 238233}, NULL) = 0
futex(0x8112054, FUTEX_WAKE, 1) = 1
futex(0x8112050, FUTEX_WAKE, 1) = 1
futex(0x805c588, FUTEX_WAKE, 1) = 1
futex(0x81aa44c, 0x4 /* FUTEX_??? */, 1) = 1
futex(0x805c488, FUTEX_WAKE, 1) = 1
futex(0x805c508, FUTEX_WAIT, 2, NULL
Process 25798 detached

the slab usage is 99.7% and not sure that is casuing any issue? and what caused this heavy slab usage.

[Rick Beldin]

The slabs are pools from which caches are allocated. High consumption by themselves isn't an indication of any specific problem. Look for caches that are full or nearly full.
You may be able to relate that back to something specific, but more likely it is falling into one of the generic caches.

As for the java strace - that looks pretty normal. Threads go to sleep and 'spin' on resources - which may be allocated within Java - until they are free. This would be what I would expect you would get in profiling almost any multithreaded app.

What problem are you trying to solve? High memory consumption? High cpu usage?

[skt_skt]

i was addressing the high CU usage.. The usage is back to normal after the clean restart of the app.Not sure what caused the problem.

[Heironimus]

Trying to use a normal system tool like strace on a JVM is usually an exercise in futility. I've seen runaway java processes many times, and it was always caused by application problems. Don't be surprised if it happens again.