JDK 1.5.0 & JDK 6.0 JVM crash executing JSVN, a Java Apps

[Sepp Stadelmann]

here I like to report how sadly JDK 1.5.0 JVM crashes !

Demonstrating what I do with jsvn, the subversion kit held in Java
we are running from JDK 1.5.0, a jsvn checkout command and run into a crash of the JVM
we have observed this first on a rx2660    / OVMS 8.3-1H1 / JDK 1.5.0
and now on our brand new         rx2800 i2 / OVMS 8.4     / JDK 1.5.0
and we got a timeout on          rx2800 i2 / OVMS 8.4     / JDK 6.0

Given the crash happens because a CRTL is not up to date - we are very willing to patch it!

The situation changes sometimes. today we see not a stack dump on screen but we get a JAVA$JAVA.DMP file
But 30 minutes later one can see stack when a application crashes.

see more comment below ...

IA64-2>jsvn info http://svn.svnkit.com/repos/svnkit
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: info
14: http://svn.svnkit.com/repos/svnkit
Path: svnkit
URL: http://svn.svnkit.com/repos/svnkit
Repository Root: http://svn.svnkit.com/repos/svnkit
Repository UUID: 0a862816-5deb-0310-9199-c792c6ae6c6e
Revision: 7864
Node Kind: directory
Last Changed Author: alex
Last Changed Rev: 7864
Last Changed Date: 2011-08-10 00:17:40 +0200 (Wed, 10 Aug 2011)

IA64-2>jsvn info http://svn.svnkit.com/repos/svnkit/trunk
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: info
14: http://svn.svnkit.com/repos/svnkit/trunk
Path: trunk
URL: http://svn.svnkit.com/repos/svnkit/trunk
Repository Root: http://svn.svnkit.com/repos/svnkit
Repository UUID: 0a862816-5deb-0310-9199-c792c6ae6c6e
Revision: 7864
Node Kind: directory
Last Changed Author: alex
Last Changed Rev: 7864
Last Changed Date: 2011-08-10 00:17:40 +0200 (Wed, 10 Aug 2011)

Here and now we aim at downloading the trunk sources for the svnkit!
It starts downloading and then it crashes with a Stack Trace
We have patched OVMS 8.4 to the latest level, but OVMS in India
tells us that this is running at theire site and we have to patch
but dont tell us what we have to patch. good service anyway.

IA64-2>jsvn co http://svn.svnkit.com/repos/svnkit/trunk trunk
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: co
14: http://svn.svnkit.com/repos/svnkit/trunk
15: trunk
A    trunk/svnkit-cli
A    trunk/svnkit-cli/src
A    trunk/svnkit-cli/src/main
A    trunk/svnkit-cli/src/main/java
A    trunk/svnkit-cli/src/main/java/org
A    trunk/svnkit-cli/src/main/java/org/tmatesoft
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCreateCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminVerifyCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCommandEnvironment.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminListTransactionsCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminDumpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRecoverCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminListLocksCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetRevPropCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminOption.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminPackCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminLoadCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdmin.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetUUIDCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminHotCopyCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRemoveTransactionsCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminUpgradeCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminHelpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRemoveLocksCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetLogCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNDumpFilter.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNCommandUtil.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNSync.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNConsoleAuthenticationProvider.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVN.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook/SVNLookHelpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook/SVNLookLockCommand.java

OpenVMS stack trace:
IA64-2>dir

Directory DKA3:[stadelma.SW-PROJEKTE.asf.svnkit]

JAVA$JAVA.DMP;1     trunk.DIR;1

Total of 2 files.
IA64-2>

Any thougths wellcome
Thanks and regards
Sepp Stadelmann

an the developer of this text entry system should fix the following:

The system closes me out at 20'000 chars. OK I can stay with that. BUT

I can take word which is able to count chars properly. As opposit to this glorry system. the text entry system here is unable to do so; or in other words it counts the HTML overhead and that way it reaches sooner 20'000 chars. ;-) Also adding my java$java.dmp file has ended in a crash ...

[Sepp Stadelmann]

OK - then let' me ask more basic

may someone can provide me a pointer to literatur covering the subject:

How to analyze a Java$Java.DMP file on a OpenVMS rx2800 i2 IA64 machine

Sepp

[H.Becker]

>>> How to analyze a Java$Java.DMP file on a OpenVMS rx2800 i2 IA64 machine

Did you try the debugger or sda, aka analyze/process <dmp-file> or analyze/system <dmp-file>?

[edit] Sorry, the latter shoud read analyze/crash for a dump file.

[Sepp Stadelmann]

IA64-2>analyze/process JAVA$JAVA.DMP

         OpenVMS I64 Debug64 Version V8.4-001

%DEBUG-W-IMAGENF, target image JAVA$JAVA not found on host system
%DEBUG-W-IMAGENF, target image JAVA$HOTSPOT_SHR not found on host system
%DEBUG-E-MAINFUNCNYI, Main Function DBG$SCAN_GST IA64 not yet implemented on this architecture
%DEBUG-I-CRMPSCFAIL, failed to map-in the Debugger Symbol Table (DST)
-DEBUG-E-BADSTATUS, bad status returned from SYS$CRMPSC
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
%DEBUG-I-CRMPSCFAIL, failed to map-in the Debugger Symbol Table (DST)
-DEBUG-E-BADSTATUS, bad status returned from SYS$CRMPSC
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
%DEBUG-I-NOUNIVERSALS, shareable image contains no universal symbols
%SYSTEM-F-IMGDMP, dynamic image dump signal at PC=0000000001134CF0, PS=0000001B
%DEBUG-I-ERRINSDEC, error decoding instruction at current PC
%DEBUG-E-INTERR, debugger error in RSTACCESS\DBG$STA_VALSPEC - Dwarf AND DST or session corruption
break on unhandled exception preceding
DBG>

That is all I get; I know I am at a debugger prompt! but what shall I do now without the many missing symbol tabvles and potentially the source code of the crashing JDK 150 JVM!

I would like to present you a stack dump as one normally gets when it has a application crashing. But I dont know why I do not get one, because few months back I got a clean stack dump to the screen for the same crashing apps.

Thanks for your help, and maybe you know how to proceeed from here

Sepp

[H.Becker]

I don't know why the debugger has some problems. Did you run the usual Java setup command procedure before analyzing the dump? At least the debugger should find the images. There is likely no debug info in these images. But at least in the shareable images there should be symbols the debugger can use to symbolize some addresses. I would try "analyze/crash" and do a "show crash".

[Sepp Stadelmann]

Too long again, this glory text entry system is a real ... and does not even allow a longer module and call stack.

Hence find it in the attachment

Sepp

[Sepp Stadelmann]

Now please find my crash dump in the attachmet due to known limitations of this HP page.

any clue which eco and patches I have to install.

any hints about what to patch is very welcome, I am not proficient on that.

OR

is this one for the engineering team java hot spot machine engineering?

Sepp

[H.Becker]

It seems the accvio (accessing va FFFFFFFFFFFFFFF5) is in librtl, which is called from the crtl, which is called from some Java rtl. That doesn't mean that there is a problem in either librtl or the crtl: the bogus va may be passed into these RTLs. It would help to know the functions on the call stack. whoever has the linker maps for these shareables can easily find the functions. In the debugger, you can try to do a "set image" to LIBRTL and DECC$SHR and for both settings repeat the "show call". That may show some more image specific details from the call stack.

On the other hand, I would check the usual problem area for Java on VMS: resources, quotas, etc. Didn't someone say it worked for her/him? To find out what the different environments are seems to be another good approach to resolve the problem.

[Sepp Stadelmann]

Thanks .-1;

I will do as you recommend!

Regarding resources for JVM - can you please check my switches shown in the initial message of this topic?

I added the -Xms<size> and the -Xmx<size> switch for the JVM yesterday but it does not make any difference.

But maybe I miss another important switch for the JVM.

And in the end --- even when I can find with your help the problem --- it is anSW engineering issue --- maybe sombody can tell me about the latest versions of this libraries to be used to avoid the crash, --- maybe sombod knows the patch for those libraries, .... if I would know sombody where that programm is fault free running, in particular the jsvn checkout command, the jsvn update command, then I could ask him to tell me about all the engaged module / library versions to check if I have the same.

But then I guess the trouble shooting from my part has to end as I can't fix the preoblem unless it can be done by an existing ECO or an existing Patch, in which case I am open to receive names of the patch or the patch to apply.

Sepp

[H.Becker]

It's not easy to find out why an incorrect VA was passed to LIB$VM_FREE. Usually this happens when some code wrote more bytes than actually allocated, that is, it overwrote some data structures for malloc/free. Java was designed not to do this. To analyze the problem someone needs the process dump and access to the current sources of librtl and decc$shr and maybe the java shareables. You should be able to give some more details from the call stack with set image to JAVA$JAVA_VMS_SHR and JAVA$JAVA_SHR. All the calls above the condition handler are not important, here. All the calls without an image name are likely in your/svnkit's java/byte code. That's all I can think of what you can do other than logging a call. Even if this turns out to be a resource problem, an ACCVIO is not the right error message to let you know about such problems. Is there any recommendation from svnkit to set the heap size? I would not increase the max without checking whether it is needed, aka whether there is much garbage collection going on. I would rather go for less heap than for more. With resources I also thought of file limit, byte limit and channelcnt, etc. Svnkit will write a lot of files (from the log it seems you checked out less than 10%). On the other hand, until the underlying problem is analyzed and fixed I would use a Linux box and svn or svnkit to check out the sources. You can copy the files to VMS or serve the directory.