JVM Crashed on its way to start

[Indranil Adak_1]

Hello,

We have a JVM Application(B) that got crashed on its way to start. Note that, this is application is spawned within another JVM Application(A). The problem is not reproducible.

The application B failed to write anything in the application log file, which made us think that it never started at all. This is also evident from the timestamp in the hs_err file generated by Java Hotspot. It means that the application crashed while being started up.

Could it happen because of less physical memory? Because, we think that, we should see a differnt message when enough physical memory is not available for starting up a JVM. We do not normally see the JVM crashing.

Any help will be greatly appreciated.

The contents of the hs_err file is given below.


Unexpected Signal : 4 occurred at PC=0x73421D80
Function=[Unknown.]
Library=(N/A)

NOTE: We are unable to locate the function name symbol for the error
just occurred. Please refer to release documentation for possible
reason and solutions.


Current Java thread:
"main" prio=8 tid=00070fc0 nid=1 lwp_id=-1 runnable [0x00000000..0x7fffe8d8]

Dynamic libraries:
/opt/java1.4/bin/IA64N/java
text:0x00001000-0x00024aa0 data:0x00025000-0x000271a8
/usr/lib/hpux32/libpthread.so.1
text:0xc008d000-0xc011fcb0 data:0x777ef000-0x777f1508
/opt/java1.4/jre/lib/IA64N/server/libjvm.so
text:0xc80c9000-0xc8dd7c90 data:0x77500000-0x776fba08
/usr/lib/hpux32/libdl.so.1
text:0xc039f000-0xc03a1660 data:0x777ec000-0x777ec068
/usr/lib/hpux32/libc.so.1
text:0xc0120000-0xc039e300 data:0x777d0000-0x777e10f8
/usr/lib/hpux32/libm.so.1
text:0xc050a000-0xc066fba0 data:0x777e9000-0x777e95d8
/usr/lib/hpux32/librt.so.1
text:0xc340d000-0xc34131f0 data:0x777e6000-0x777e61a8
/usr/lib/hpux32/libuca.so.1
text:0xc0d44000-0xc0d4baf0 data:0x777e5000-0x777e5150
/usr/lib/hpux32/libunwind.so.1
text:0xc0cfd000-0xc0d43470 data:0x777ce000-0x777cf1f0
/usr/lib/hpux32/libcl.so.1
text:0xc3178000-0xc3178304 data:0x777e2000-0x777e2018
/usr/lib/hpux32/libCsup.so.1
text:0xc0cb5000-0xc0cfc0f0 data:0x777c7000-0x777cdbe8
/usr/lib/hpux32/libIO77.so.1
text:0xc3221000-0xc32bd410 data:0x777b8000-0x777c2918
/opt/java1.4/jre/lib/IA64N/native_threads/libhpi.so
text:0xc51e5000-0xc5208440 data:0x777ae000-0x777af880
/opt/java1.4/jre/lib/IA64N/libverify.so
text:0xc5209000-0xc5227a10 data:0x777aa000-0x777ab0a8
/opt/java1.4/jre/lib/IA64N/libjava.so
text:0xc5228000-0xc526e530 data:0x777a8000-0x777a95d0
/opt/java1.4/jre/lib/IA64N/libzip.so
text:0xc526f000-0xc529d250 data:0x777a4000-0x777a5870

Heap at VM Abort:
Heap
def new generation total 5632K, used 181K [6b400000, 6ba10000, 6c950000)
eden space 5056K, 3% used [6b400000, 6b42d728, 6b8f0000)
from space 576K, 0% used [6b8f0000, 6b8f0000, 6b980000)
to space 576K, 0% used [6b980000, 6b980000, 6ba10000)
tenured generation total 12480K, used 0K [6c950000, 6d580000, 6f400000)
the space 12480K, 0% used [6c950000, 6c950000, 6c950200, 6d580000)
compacting perm gen total 16384K, used 1049K [6f400000, 70400000, 73400000)
the space 16384K, 6% used [6f400000, 6f506670, 6f506800, 70400000)

Local Time = Thu Dec 29 17:14:02 2005
Elapsed Time = 5
#
# HotSpot Virtual Machine Error : 4
# Please report this error to HP customer support.
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2 1.4.2.07-050121-21:06-IA64N IA64 mixed mode)
#

I have not been able to attach the core file here because of the size restriction. I can send it in an e-mail to the responder.

Thanks in advance
Indranil

[RAC_1]

Without a clear error messge, it is very difficult to troubleshoot. The error signal 4 means illegal instuction. Aprt from what you see, do you any error messages at the time of application startup failure?? Check dmesg/syslog.log. Also check cpu,disk,swap,memeory utlization at the time of this error.

Last but not least, are you up to date on java patches??

[Arunvijai_4]

Hello,

What is your Java version and patch level of your server ? You can grab the latest version from,www.hp.com/go/java and install the prescribed patcheds as well.

-Arun

[Indranil Adak_1]

Hello,

There were no error messsages apparently available in the syslog.

As this problem was discovered after a certain period of time, we do not have the system level statistics.

Yes, we have already checked the Java versions and recommended patches. We are running JRE 1.4.2_07 with all the recommended patches available.

Regards
Indranil

[Arunvijai_4]

Hello,

How do you invoke JVM in A and B ? Are there any parameters with comman option "java -D " ?

-Arun

[Indranil Adak_1]

Hello,

We do not use any Java specific parameter using -D. We use an application specific property though.

e.g. java -D=

Please let me know if anyone is interested in debugging the core file. I can send it to the responder's e-mail a/c.

Regards
Indranil

[Steven E. Protter]

Shalom,

General recommendations.

http://www.hp.com/go/java

Install all required and recommended patches and their dependencies, which you get from the patch database:
http://itrc.hp.com

You may be able to do strings core to show the readable characters in the core file and upload that file.

HP support can allow you to email up to 5 MB and if you need to post a bigger file, they will set up a temporary ftp site for you.

SEP

[Chew Chye Guan]

On this incident, this is what I gathered on the system situations....

There was a MCA problem reported by /var/adm/crash. I will appended part of the info.

-rwxr-xr-x 1 root root 1 Jan 17 17:49 bounds
drwxr-xr-x 2 root root 8192 Jan 17 17:52 crash.6

There is some hardware problems but what is not sure where it is about 30mins after the JVM crash ie JVM crash around 17:14. And bounds and system dump at 17:49-17-52...

In the INDEX file, it shows...comment savecrash crash dump INDEX file shows the top few lines as:....

version 2
hostname atco2-0
modelname ia64 hp server rx2600
panic MCA, IIP:0xe00000000044b260 IFA:0x0000000000695140
dumptime 1137487527 Tue Jan 17 17:45:27 JST 2006
savetime 1137487762 Tue Jan 17 17:49:22 JST 2006
release @(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38
PDT 2003 $
memsize 8578154496
chunksize 268435456
defcompchunk 0x0000000000000000

virtual to physical information for IA64
registers 0x8a899a0
vhpt 0xe000000140000000 2097152
itr 0x8aa31c0 63
dtr 0x8eb9580 74

....
Clearly a MCA error..(hardware).

Now the question that I had, is this possible? ie JVM dumps first and then half an hr later, the system crashed.

Are they related?
thanks
chyeguan
(working with Indranil on the same project)

[Chew Chye Guan]

oops ignore my previous submission. It was having the wrong date.. Crashed was 17 Jan. Indranil reported problem was Dec 29...

apologize.

[Chew Chye Guan]

ops. looks like I got the wrong date..
Indranil's problem occured on 29 Dec 17:14 where else the MCA problem occurs on 17 Jan 17:42. apologize...