JVM Crashes under Tomcat application

[Kursat Tuncel_1]

Hi,

We have been developing a tomcat application for production. Upto now, we have been doing tests on Linux, Solaris and Windows without problem. Our hardware targetted was HP-UX Itanium, so when they arrived and installed, we had a chance to deploy our application for the first time and JVM crashed immediately after loading the java code.


Here is the stack trace of error:

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32744 bytes for ChunkPool::allocate. Out of swap space?
Possible causes:
- not enough swap space left, or
- kernel parameter MAXDSIZ is very small.
#
# Internal Error (allocation.cpp:128), pid=16830, tid=23
# Error: ChunkPool::allocate
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16-jre1.6.0.03-rc2 mixed mode hp-ux-ia64)
# Please report this error to HP customer support.
#

--------------- T H R E A D ---------------

Current thread (6000000000505c00): JavaThread "CompilerThread0" daemon [_thread_in_native, id=23, lwp_id=275711, stack(9fffffff1ce01000,9fffffff1d001000)]

Stack: [9fffffff1ce01000,9fffffff1d001000]

Current CompileTask:
C3: 30% org.springframework.asm.ClassReader.accept(Lorg/springframework/asm/ClassVisitor;[Lorg/springframework/asm/Attribute;Z)V @ 2373 (4725 bytes)


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
600000000092c200 JavaThread "TP-Monitor" daemon [_thread_blocked, id=55, lwp_id=275854, stack(9fffffff19281000,9fffffff19301000)]
60000000008f4000 JavaThread "TP-Processor4" daemon [_thread_in_native, id=54, lwp_id=275853, stack(9fffffff19481000,9fffffff19501000)]
60000000008d3400 JavaThread "TP-Processor3" daemon [_thread_blocked, id=53, lwp_id=275852, stack(9fffffff19681000,9fffffff19701000)]
6000000000889400 JavaThread "TP-Processor2" daemon [_thread_blocked, id=52, lwp_id=275851, stack(9fffffff19881000,9fffffff19901000)]
6000000000870200 JavaThread "TP-Processor1" daemon [_thread_blocked, id=51, lwp_id=275850, stack(9fffffff19a81000,9fffffff19b01000)]
6000000000849200 JavaThread "http-80-Acceptor-0" daemon [_thread_in_native, id=50, lwp_id=275849, stack(9fffffff19c81000,9fffffff19d01000)]
600000000075f200 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=49, lwp_id=275848, stack(9fffffff19e81000,9fffffff19f01000)]
60000000074df800 JavaThread "QuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=47, lwp_id=275748, stack(9fffffff1a281000,9fffffff1a301000)]
600000000665f600 JavaThread "QuartzScheduler_Worker-10" [_thread_blocked, id=46, lwp_id=275747, stack(9fffffff1a481000,9fffffff1a501000)]
60000000064f9200 JavaThread "QuartzScheduler_Worker-9" [_thread_blocked, id=45, lwp_id=275746, stack(9fffffff1a681000,9fffffff1a701000)]
6000000004dbfc00 JavaThread "QuartzScheduler_Worker-8" [_thread_blocked, id=44, lwp_id=275745, stack(9fffffff1a881000,9fffffff1a901000)]
6000000004d14000 JavaThread "QuartzScheduler_Worker-7" [_thread_blocked, id=43, lwp_id=275744, stack(9fffffff1aa81000,9fffffff1ab01000)]
6000000004c1f200 JavaThread "QuartzScheduler_Worker-6" [_thread_blocked, id=42, lwp_id=275743, stack(9fffffff1ac81000,9fffffff1ad01000)]
6000000004875200 JavaThread "QuartzScheduler_Worker-5" [_thread_blocked, id=41, lwp_id=275742, stack(9fffffff1ae81000,9fffffff1af01000)]
6000000004777000 JavaThread "QuartzScheduler_Worker-4" [_thread_blocked, id=40, lwp_id=275741, stack(9fffffff1b081000,9fffffff1b101000)]
600000000463f600 JavaThread "QuartzScheduler_Worker-3" [_thread_blocked, id=39, lwp_id=275740, stack(9fffffff1b281000,9fffffff1b301000)]
600000000463c200 JavaThread "QuartzScheduler_Worker-2" [_thread_blocked, id=38, lwp_id=275739, stack(9fffffff1b481000,9fffffff1b501000)]
6000000003e7fc00 JavaThread "QuartzScheduler_Worker-1" [_thread_blocked, id=37, lwp_id=275738, stack(9fffffffba081000,9fffffffba101000)]
6000000001c3f800 JavaThread "Thread-10" [_thread_blocked, id=36, lwp_id=275733, stack(9fffffffba281000,9fffffffba301000)]
60000000015e4200 JavaThread "Thread-9" [_thread_blocked, id=35, lwp_id=275731, stack(9fffffffba481000,9fffffffba501000)]
60000000015a9a00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=34, lwp_id=275729, stack(9fffffff1b681000,9fffffff1b701000)]
600000000159a800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=33, lwp_id=275728, stack(9fffffff1b881000,9fffffff1b901000)]
600000000067c800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=32, lwp_id=275727, stack(9fffffff1ba81000,9fffffff1bb01000)]
600000000057c600 JavaThread "Timer-1" daemon [_thread_blocked, id=31, lwp_id=275726, stack(9fffffffba681000,9fffffffba701000)]
6000000001526e00 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=30, lwp_id=275723, stack(9fffffff1bc81000,9fffffff1bd01000)]
600000000123f400 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=29, lwp_id=275722, stack(9fffffff1be81000,9fffffff1bf01000)]
60000000008d4200 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=28, lwp_id=275721, stack(9fffffff1c081000,9fffffff1c101000)]
60000000007fea00 JavaThread "Timer-0" daemon [_thread_blocked, id=27, lwp_id=275720, stack(9fffffffba881000,9fffffffba901000)]
6000000000512600 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=26, lwp_id=275714, stack(9fffffff1c281000,9fffffff1c301000)]
6000000000509000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=24, lwp_id=275712, stack(9fffffff1c601000,9fffffff1c801000)]
=>6000000000505c00 JavaThread "CompilerThread0" daemon [_thread_in_native, id=23, lwp_id=275711, stack(9fffffff1ce01000,9fffffff1d001000)]
60000000004b9000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=22, lwp_id=275710, stack(9fffffff1d281000,9fffffff1d301000)]
60000000004b3e00 JavaThread "Finalizer" daemon [_thread_blocked, id=21, lwp_id=275709, stack(9fffffff1d481000,9fffffff1d501000)]
60000000004b1600 JavaThread "Reference Handler" daemon [_thread_blocked, id=20, lwp_id=275708, stack(9fffffff1d681000,9fffffff1d701000)]
6000000000059a00 JavaThread "main" [_thread_in_native, id=1, lwp_id=-1, stack(9fffffffffdff000,a000000000000000)]

Other Threads:
60000000001df400 VMThread [stack: 9fffffff1d881000,9fffffff1d901000] [id=19, lwp_id=275707]
600000000006e800 WatcherThread [stack: 9fffffff1ca81000,9fffffff1cb01000] [id=25, lwp_id=275713]

VM state: not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 299328K, used 61267K [9fffffff69eb0000, 9fffffff7f400000, 9fffffff7f400000)
eden space 255680K, 6% used [9fffffff69eb0000,9fffffff6afe8858,9fffffff79860000)
from space 43648K, 99% used [9fffffff79ec0000,9fffffff7c95c4b0,9fffffff7c960000)
to space 43648K, 0% used [9fffffff7c960000,9fffffff7c960000,9fffffff7f400000)
PSOldGen total 699072K, used 11472K [9fffffff3f400000, 9fffffff69eb0000, 9fffffff69eb0000)
object space 699072K, 1% used [9fffffff3f400000,9fffffff3ff343e0,9fffffff69eb0000)
PSPermGen total 524288K, used 70536K [9fffffff1f400000, 9fffffff3f400000, 9fffffff3f400000)
object space 524288K, 13% used [9fffffff1f400000,9fffffff238e2040,9fffffff3f400000)

Dynamic libraries:
/opt/java6/bin/IA64W/java
text:0x4000000000000000-0x4000000000019a50 data:0x6000000000000000-0x6000000000000520
/usr/lib/hpux64/libpthread.so.1
text:0xc0000000000a4000-0xc0000000001ad4c0 data:0x9fffffffbf7ec000-0x9fffffffbf7efcc0
/opt/java6/bin/IA64W/../../jre/lib/IA64W/jli//libjli.so
text:0xc000000007900000-0xc00000000791dcc0 data:0x9fffffffbf7ea000-0x9fffffffbf7eb4c0
/usr/lib/hpux64/libdl.so.1
text:0xc00000000045c000-0xc000000000460470 data:0x9fffffffbf7e9000-0x9fffffffbf7e9160
/usr/lib/hpux64/libc.so.1
text:0xc0000000001b0000-0xc000000000458b40 data:0x9fffffffbf7d3000-0x9fffffffbf7e8ed8
/opt/java6/jre/lib/IA64W/server/libjvm.so
text:0xc000000007da0000-0xc000000009ca5ce0 data:0x9fffffffbf4bd000-0x9fffffffbf7b0118
/usr/lib/hpux64/libm.so.1
text:0xc0000000006d0000-0xc000000000829ce0 data:0x9fffffffbf7d2000-0x9fffffffbf7d2c88
/usr/lib/hpux64/librt.so.1
text:0xc000000000ef4000-0xc000000000efb580 data:0x9fffffffbf7c5000-0x9fffffffbf7c50e0
/usr/lib/hpux64/libuca.so.1
text:0xc0000000008b4000-0xc0000000008bc020 data:0x9fffffffbf7c4000-0x9fffffffbf7c4150
/usr/lib/hpux64/libelf.so.1
text:0xc000000007920000-0xc0000000079498e0 data:0x9fffffffbf7bd000-0x9fffffffbf7bd9b0
/usr/lib/hpux64/libunwind.so.1
text:0xc00000000082c000-0xc000000000881ed0 data:0x9fffffffbf7bc000-0x9fffffffbf7bc9d8
/usr/lib/hpux64/libstd_v2.so.1
text:0xc0000000004c8000-0xc000000000671180 data:0x9fffffffbf4ac000-0x9fffffffbf4b9468
/usr/lib/hpux64/libCsup.so.1
text:0xc000000000674000-0xc0000000006cc4a0 data:0x9fffffffbf4a4000-0x9fffffffbf4aa578
/opt/java6/jre/lib/IA64W/native_threads/libhpi.so
text:0xc00000000794c000-0xc0000000079654d0 data:0x9fffffffbf4a2000-0x9fffffffbf4a3988
/usr/lib/hpux64/libhg.so
text:0xc000000000f00000-0xc000000000f02f00 data:0x9fffffffbf4bc000-0x9fffffffbf4bc0b8
/opt/java6/jre/lib/IA64W/libverify.so
text:0xc000000007968000-0xc000000007989ac0 data:0x9fffffffbf498000-0x9fffffffbf499220
/opt/java6/jre/lib/IA64W/libjava.so
text:0xc00000000798c000-0xc0000000079f5d90 data:0x9fffffffbf490000-0x9fffffffbf493658
/opt/java6/jre/lib/IA64W/libzip.so
text:0xc0000000079f8000-0xc000000007a2d6b0 data:0x9fffffffbf48c000-0x9fffffffbf48d8a0
/opt/java6/jre/lib/IA64W/libnet.so
text:0xc000000007a30000-0xc000000007a5f970 data:0x9fffffffbf4ab000-0x9fffffffbf4abae8
/usr/lib/hpux64/libnss_dns.so.1
text:0xc000000001050000-0xc00000000105c120 data:0x9fffffffbf4a1000-0x9fffffffbf4a1338
/usr/lib/hpux64/libnss_nis.so.1
text:0xc000000000ee4000-0xc000000000ef3250 data:0x9fffffffbf489000-0x9fffffffbf489ad8
/usr/lib/hpux64/libnsl.so.1
text:0xc000000000bec000-0xc000000000ce08a0 data:0x9fffffffbf432000-0x9fffffffbf43dc78
/usr/lib/hpux64/libxti.so.1
text:0xc000000000884000-0xc0000000008b3510 data:0x9fffffffbf42e000-0x9fffffffbf4309a0
/usr/lib/hpux64/libnss_files.so.1
text:0xc0000000008c0000-0xc0000000008d4f70 data:0x9fffffffbf488000-0x9fffffffbf488c80
/opt/java6/jre/lib/IA64W/libmanagement.so
text:0xc000000000f04000-0xc000000000f0fe20 data:0x9fffffffbf481000-0x9fffffffbf481360

VM Arguments:
jvm_args: -Xmx1024m -Xms1024m -XX:PermSize=512m -XX:MaxPermSize=512m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/tomcatDeploy/conf/logging.properties -Djava.endorsed.dirs=/usr/local/tomcat/endorsed -Dcatalina.base=/var/tomcatDeploy -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/var/tomcatDeploy/temp
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/opt/java6
PATH=/usr/sbin:/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/usr/contrib/Q4/bin:/opt/perl/bin:/opt/ipf/bin:/opt/gvsd/bin:/opt/nettladm/bin:/opt/fcms/bin:/opt/wbem/bin:/opt/wbem/sbin:/opt/sas/bin:/opt/graphics/common/bin:/opt/atok/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/sec_mgmt/bastille/bin:/opt/caliper/bin:/opt/resmon/bin:/opt/drd/bin:/opt/dsau/bin:/opt/dsau/sbin:/opt/firefox:/opt/gnome/bin:/usr/contrib/kwdb/bin:/opt/mozilla:/opt/perl_32/bin:/opt/perl_64/bin:/opt/prm/bin:/opt/sfm/bin:/opt/swm/bin:/opt/sec_mgmt/spc/bin:/opt/ssh/bin:/opt/sentinel/bin:/opt/swa/bin:/opt/hpsmh/bin:/opt/thunderbird:/opt/langtools/bin:/opt/gwlm/bin:/opt/ignite/bin:/opt/cfg2html:/sbin:/home/root:/usr/local/bin
LD_LIBRARY_PATH=/opt/java6/jre/lib/IA64W:/opt/java6/jre/lib/IA64W/server:/opt/java6/jre/../lib/IA64W
SHELL=/sbin/sh
PTHREAD_COMPAT_MODE=0

Signal Handlers:
SIGSEGV: [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGBUS : [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGFPE : [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGPIPE: [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGXFSZ: [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGILL : [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGUSR2: [libjvm.so+0xffdf5c18], sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP : [libjvm.so+0xffdf2d28], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGINT : SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0xffdf2d28], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGTERM: [libjvm.so+0xffdf2d28], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGRTMIN: [libjvm.so+0xffdf5de8], sa_mask[0]=0x00000000, sa_flags=0x00000010
SIGRTMIN+1: [libjvm.so+0xffdf72e8], sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050


--------------- S Y S T E M ---------------

OS:HP-UX
uname:HP-UX B.11.31 U ia64
rlimit: STACK 1048576k, CORE 2097151k, NOFILE 4096, AS infinity
load average:2.00 1.22 0.72

CPU:total 16

Memory: 4k page, physical 268299144k(228581256k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (11.0-b16-jre1.6.0.03-rc2) for hp-ux-ia64 JRE (1.6.0), built on Feb 6 2009 05:46:46 by "jinteg" with aCC

time: Sat May 2 19:17:45 2009
elapsed time: 255 seconds

Can anyone suggest anything over it?

I also opened ticket to local HP support, but no response yet.

Thank you in advance.

[Kursat Tuncel_1]

There has been an update. This problem only happens under JDK 1.6.003. When using JDK 1.6.002, application works fine, no core dumps. Anybody using JDK 1.6.003 for HP-UX B.11.31 should carefully investigate coredump issues.

[Steven E. Protter]

Shalom,

Most of the time, people fix this by updating and installing recommended patches from http://www.hp.com/go/java

SEP

[Kursat Tuncel_1]

Hi,

I indeed applied patches recommended by HP.

http://docs.hp.com/en/HPUXJAVAPATCHES/index.html

quote
"
HP-UX 11.31 patches required for Javaâ ¢ 6.0 with the Sept '08 Quality Pack
This list of patches is for Java 6.0 running on HP-UX 11.31 with the Sept '08 Quality Pack installed.

The table shows the earliest patch, as well as patches that supersede it. The earliest patch or any patch in the superseded list is acceptable to resolve the problem affecting Java. Note that as a result, some required patches may appear as supersedes of other required patches. Icons and headings are explained below.
"

My System:
QPKBASE B.11.31.0809.326 Base Quality Pack Bundle for HP-UX 11i v3, September 2008

Applied Patches:
(Latest, Supedseeding patches)
PHCO_38837
PHSS_39102

Application crashes JVM in 1.6.003.
Application works perfectly in 1.6.002.

After reading deeply release notes of 1.6.003, I realized that this version is somewhat introduces major changes such as using IPV6 stack by default. I already set it back to use ipv4 stack.

Definitely, there is definitly memory leak in 1.6.003, in compiler thread. I don't know if this bug is hit by me first but my application uses springframework intensively. Stack trace points to a class related to springframework.

[Dennis Handly]

What was your maxdsiz_64bit? 4 Gb? Large enough to show it is a leak?