JDK 1.5.0 & JDK 6.0 JVM crash executing JSVN, a Java Apps

[Sepp Stadelmann]

here I like to report how sadly JDK 1.5.0 JVM crashes !

Demonstrating what I do with jsvn, the subversion kit held in Java
we are running from JDK 1.5.0, a jsvn checkout command and run into a crash of the JVM
we have observed this first on a rx2660    / OVMS 8.3-1H1 / JDK 1.5.0
and now on our brand new         rx2800 i2 / OVMS 8.4     / JDK 1.5.0
and we got a timeout on          rx2800 i2 / OVMS 8.4     / JDK 6.0

Given the crash happens because a CRTL is not up to date - we are very willing to patch it!

The situation changes sometimes. today we see not a stack dump on screen but we get a JAVA$JAVA.DMP file
But 30 minutes later one can see stack when a application crashes.

see more comment below ...

IA64-2>jsvn info http://svn.svnkit.com/repos/svnkit
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: info
14: http://svn.svnkit.com/repos/svnkit
Path: svnkit
URL: http://svn.svnkit.com/repos/svnkit
Repository Root: http://svn.svnkit.com/repos/svnkit
Repository UUID: 0a862816-5deb-0310-9199-c792c6ae6c6e
Revision: 7864
Node Kind: directory
Last Changed Author: alex
Last Changed Rev: 7864
Last Changed Date: 2011-08-10 00:17:40 +0200 (Wed, 10 Aug 2011)

IA64-2>jsvn info http://svn.svnkit.com/repos/svnkit/trunk
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: info
14: http://svn.svnkit.com/repos/svnkit/trunk
Path: trunk
URL: http://svn.svnkit.com/repos/svnkit/trunk
Repository Root: http://svn.svnkit.com/repos/svnkit
Repository UUID: 0a862816-5deb-0310-9199-c792c6ae6c6e
Revision: 7864
Node Kind: directory
Last Changed Author: alex
Last Changed Rev: 7864
Last Changed Date: 2011-08-10 00:17:40 +0200 (Wed, 10 Aug 2011)

Here and now we aim at downloading the trunk sources for the svnkit!
It starts downloading and then it crashes with a Stack Trace
We have patched OVMS 8.4 to the latest level, but OVMS in India
tells us that this is running at theire site and we have to patch
but dont tell us what we have to patch. good service anyway.

IA64-2>jsvn co http://svn.svnkit.com/repos/svnkit/trunk trunk
0: DKA3:[JAVA$150.BIN]JAVA$JAVA.EXE;1
1: -Xnocatch
2: -Xms54m
3: -Xmx1024m
4: -Djava.util.logging.config.file=/svnkit_home/logging.properties
5: -Dhttp.auth.preference=Basic
6: -Dhttp.proxyhost=proxy.ch.winterthur.com
7: -Dhttp.proxyport=8080
8: -Dhttp.proxyuser=C770817
9: -Dhttp.proxypassword=gagapw01
10: -cp
11: /svnkit_home/svnkit.jar:/svnkit_home/svnkit-cli.jar:/svnkit_home/trilead.jar:/svnkit_home/jna.jar:/svnkit_home/sqljet.1.0.1.jar:
/svnkit_home/antlr-runtime-3.1.3.jar
12: org.tmatesoft.svn.cli.svn.SVN
13: co
14: http://svn.svnkit.com/repos/svnkit/trunk
15: trunk
A    trunk/svnkit-cli
A    trunk/svnkit-cli/src
A    trunk/svnkit-cli/src/main
A    trunk/svnkit-cli/src/main/java
A    trunk/svnkit-cli/src/main/java/org
A    trunk/svnkit-cli/src/main/java/org/tmatesoft
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCreateCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminVerifyCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCommandEnvironment.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminListTransactionsCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminDumpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRecoverCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminListLocksCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetRevPropCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminOption.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminPackCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminLoadCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdmin.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetUUIDCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminHotCopyCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRemoveTransactionsCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminUpgradeCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminHelpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminRemoveLocksCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnadmin/SVNAdminSetLogCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNDumpFilter.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNCommandUtil.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNSync.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVNConsoleAuthenticationProvider.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/SVN.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook/SVNLookHelpCommand.java
A    trunk/svnkit-cli/src/main/java/org/tmatesoft/svn/cli/svnlook/SVNLookLockCommand.java

OpenVMS stack trace:
IA64-2>dir

Directory DKA3:[stadelma.SW-PROJEKTE.asf.svnkit]

JAVA$JAVA.DMP;1     trunk.DIR;1

Total of 2 files.
IA64-2>

Any thougths wellcome
Thanks and regards
Sepp Stadelmann

an the developer of this text entry system should fix the following:

The system closes me out at 20'000 chars. OK I can stay with that. BUT

I can take word which is able to count chars properly. As opposit to this glorry system. the text entry system here is unable to do so; or in other words it counts the HTML overhead and that way it reaches sooner 20'000 chars. ;-) Also adding my java$java.dmp file has ended in a crash ...

[Sepp Stadelmann]

OK - then let' me ask more basic

may someone can provide me a pointer to literatur covering the subject:

How to analyze a Java$Java.DMP file on a OpenVMS rx2800 i2 IA64 machine

Sepp

[H.Becker]

>>> How to analyze a Java$Java.DMP file on a OpenVMS rx2800 i2 IA64 machine

Did you try the debugger or sda, aka analyze/process <dmp-file> or analyze/system <dmp-file>?

[edit] Sorry, the latter shoud read analyze/crash for a dump file.

[Sepp Stadelmann]

IA64-2>analyze/process JAVA$JAVA.DMP

         OpenVMS I64 Debug64 Version V8.4-001

%DEBUG-W-IMAGENF, target image JAVA$JAVA not found on host system
%DEBUG-W-IMAGENF, target image JAVA$HOTSPOT_SHR not found on host system
%DEBUG-E-MAINFUNCNYI, Main Function DBG$SCAN_GST IA64 not yet implemented on this architecture
%DEBUG-I-CRMPSCFAIL, failed to map-in the Debugger Symbol Table (DST)
-DEBUG-E-BADSTATUS, bad status returned from SYS$CRMPSC
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
%DEBUG-I-CRMPSCFAIL, failed to map-in the Debugger Symbol Table (DST)
-DEBUG-E-BADSTATUS, bad status returned from SYS$CRMPSC
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
%DEBUG-I-NOUNIVERSALS, shareable image contains no universal symbols
%SYSTEM-F-IMGDMP, dynamic image dump signal at PC=0000000001134CF0, PS=0000001B
%DEBUG-I-ERRINSDEC, error decoding instruction at current PC
%DEBUG-E-INTERR, debugger error in RSTACCESS\DBG$STA_VALSPEC - Dwarf AND DST or session corruption
break on unhandled exception preceding
DBG>

That is all I get; I know I am at a debugger prompt! but what shall I do now without the many missing symbol tabvles and potentially the source code of the crashing JDK 150 JVM!

I would like to present you a stack dump as one normally gets when it has a application crashing. But I dont know why I do not get one, because few months back I got a clean stack dump to the screen for the same crashing apps.

Thanks for your help, and maybe you know how to proceeed from here

Sepp

[H.Becker]

I don't know why the debugger has some problems. Did you run the usual Java setup command procedure before analyzing the dump? At least the debugger should find the images. There is likely no debug info in these images. But at least in the shareable images there should be symbols the debugger can use to symbolize some addresses. I would try "analyze/crash" and do a "show crash".

[Sepp Stadelmann]

Too long again, this glory text entry system is a real ... and does not even allow a longer module and call stack.

Hence find it in the attachment

Sepp

[Sepp Stadelmann]

Now please find my crash dump in the attachmet due to known limitations of this HP page.

any clue which eco and patches I have to install.

any hints about what to patch is very welcome, I am not proficient on that.

OR

is this one for the engineering team java hot spot machine engineering?

Sepp

[H.Becker]

It seems the accvio (accessing va FFFFFFFFFFFFFFF5) is in librtl, which is called from the crtl, which is called from some Java rtl. That doesn't mean that there is a problem in either librtl or the crtl: the bogus va may be passed into these RTLs. It would help to know the functions on the call stack. whoever has the linker maps for these shareables can easily find the functions. In the debugger, you can try to do a "set image" to LIBRTL and DECC$SHR and for both settings repeat the "show call". That may show some more image specific details from the call stack.

On the other hand, I would check the usual problem area for Java on VMS: resources, quotas, etc. Didn't someone say it worked for her/him? To find out what the different environments are seems to be another good approach to resolve the problem.

[Sepp Stadelmann]

Thanks .-1;

I will do as you recommend!

Regarding resources for JVM - can you please check my switches shown in the initial message of this topic?

I added the -Xms<size> and the -Xmx<size> switch for the JVM yesterday but it does not make any difference.

But maybe I miss another important switch for the JVM.

And in the end --- even when I can find with your help the problem --- it is anSW engineering issue --- maybe sombody can tell me about the latest versions of this libraries to be used to avoid the crash, --- maybe sombod knows the patch for those libraries, .... if I would know sombody where that programm is fault free running, in particular the jsvn checkout command, the jsvn update command, then I could ask him to tell me about all the engaged module / library versions to check if I have the same.

But then I guess the trouble shooting from my part has to end as I can't fix the preoblem unless it can be done by an existing ECO or an existing Patch, in which case I am open to receive names of the patch or the patch to apply.

Sepp

[H.Becker]

It's not easy to find out why an incorrect VA was passed to LIB$VM_FREE. Usually this happens when some code wrote more bytes than actually allocated, that is, it overwrote some data structures for malloc/free. Java was designed not to do this. To analyze the problem someone needs the process dump and access to the current sources of librtl and decc$shr and maybe the java shareables. You should be able to give some more details from the call stack with set image to JAVA$JAVA_VMS_SHR and JAVA$JAVA_SHR. All the calls above the condition handler are not important, here. All the calls without an image name are likely in your/svnkit's java/byte code. That's all I can think of what you can do other than logging a call. Even if this turns out to be a resource problem, an ACCVIO is not the right error message to let you know about such problems. Is there any recommendation from svnkit to set the heap size? I would not increase the max without checking whether it is needed, aka whether there is much garbage collection going on. I would rather go for less heap than for more. With resources I also thought of file limit, byte limit and channelcnt, etc. Svnkit will write a lot of files (from the log it seems you checked out less than 10%). On the other hand, until the underlying problem is analyzed and fixed I would use a Linux box and svn or svnkit to check out the sources. You can copy the files to VMS or serve the directory.

[Sepp Stadelmann]

Hi

I have applied some more patches to the system, but no big break through ...

So I retried again my jsvn checkout command and in the attachment I can show you some more insigth.

And I fully agree with you;

The onse with the LIBRTL source code should look at it !!! ----> HP OpenVMS Engineering

Thank you very much to help me speeding up this process.

Sepp

[Ph Vouters]

Would running Java under the debugger control help diagnose the real problem ? For more information about running Java under the debugger control, refer to http://vouters.dyndns.org/tima/OpenVMS-Java-V6.0-A_user_experience_with_OpenVMS_IA64_Java_V6.0.html (see temporary workaround for problem one).

I seem to recollect JSVN is dependant upon JNA.  Do you confirm this ? If yes, there are lots of causes for a JVM crash !

If this can help I should be glad.

Philippe

[Toine_1]

Hi,

Did you set the logical names below before you run jsvn?

$!
$ define/job/nolog decc$argv_parse_style enable
$ define/job/nolog decc$efs_case_preserve enable

And make sure the logical name DECC$EFS_CASE_SPECIAL is not defined.

$!
$ if f$edit(f$trnlnm("DECC$EFS_CASE_SPECIAL"),"UPCASE") .eqs. "ENABLE"
$ then
$    write sys$output " "
$    write sys$output "JSVNSETUP: DECC$EFS_CASE_SPECIAL defined! Please deassign logical otherwise jsvn might not function properly!
$    write sys$output " "
$ endif

We are using JSVN on OpenVMS 8.3-1H1 and 8.4 and it works fine with Java 1.5 and 1.6.

Regards,

/Toine

[Sepp Stadelmann]

this are my DECC logicals AS SET AFTER A CRASH

A    trunk6/CHANGES.txt
A    trunk6/svnkit-dav
OpenVMS stack trace:

IA64-2>show  log decc*

(LNM$PROCESS_TABLE)

  "DECC$EFS_CHARSET" = "true"
  "DECC$FD_LOCKING" = "1"

(LNM$JOB_B16ECB00)

  "DECC$ARGV_PARSE_STYLE" = "enable"
  "DECC$EFS_CASE_PRESERVE" = "enable"

(SERVER_24CCCC81)

(LNM$GROUP_000201)

(LNM$SYSTEM_TABLE)

  "DECC$CRTLMAP" = "SYS$SHARE:DECC$SHR"
  "DECC$SHR_AV" = "DECC$SHR"

(LNM$SYSCLUSTER_TABLE)

(DECW$LOGICAL_NAMES)
IA64-2>

And now I would like to know form thouse of you which say that JSVN runs if JSVN and the given version of OpenVMS is able to checkout a very large source repository at once.

I would like to know in this case the exact patch leven and the version of JAVA / JVM they use.

please do me a favor and deposit the installed products here. my onces are.

IA64-2>product show hist
------------------------------------ ----------- ----------- --- -----------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- -----------
HP I64VMS TCPIP_NTP_PAT V5.7-ECO1B   Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_DEBUG V1.0          Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_FIBRE_SCSI V1.0     Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_ICAP V1.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_LAN V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_IPC V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_MUP V2.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_MUP V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_PCSI V2.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_PCSI V1.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_RMS V2.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_RMS V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_SYS V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V3.0         Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V2.0         Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V1.0         Patch       Install     Val 11-AUG-2011
HP I64VMS GNV V2.1-3                 Full LP     Install     Val 10-AUG-2011
HP I64VMS GNV V2.1-3                 Full LP     Remove       -  10-AUG-2011
HP I64VMS VMS84I_DEBUG V1.0          Patch       Install     Val 08-AUG-2011
HP I64VMS C V7.3-19                  Full LP     Install     Val 08-AUG-2011
HP I64VMS C V7.3-18                  Full LP     Remove       -  08-AUG-2011
HP I64VMS VMS84I_LAN V1.0            Patch       Install     Val 08-AUG-2011
HP I64VMS LTT V4.13-0                Full LP     Install     Val 03-AUG-2011
HP I64VMS VMS84I_FIBRE_SCSI V1.0     Patch       Install     Val 22-JUL-2011
HP I64VMS LTT V4.13-0                Full LP     Install     Val 14-JUL-2011
HP I64VMS IDESERVER V6.52            Full LP     Install     Val 12-JUL-2011
HP I64VMS CSWS_PHP V2.20             Full LP     Install     Val 06-JUL-2011
HP I64VMS CSWS_PERL V2.1             Full LP     Install     (U) 06-JUL-2011
HP IA64VMS WSIT V3.0                 Full LP     Install     (U) 06-JUL-2011
HP I64VMS GNV V2.1-3                 Full LP     Install     Val 05-JUL-2011
HP I64VMS CSWS_JAVA V3.1             Full LP     Install     Val 05-JUL-2011
HP I64VMS CSWS V2.2                  Full LP     Install     Val 05-JUL-2011
HP I64VMS JAVA150 V1.5-5             Full LP     Install     Val 05-JUL-2011
HP I64VMS JAVA60 V1.6-2_P1           Full LP     Install     Val 05-JUL-2011
HP I64VMS PERL586_UPDATE V2.0        Patch       Install     Val 09-MAY-2011
HP I64VMS PERL V5.8-6                Full LP     Install     (U) 09-MAY-2011
IBM I64VMS WMQCLIENT V6.0            Full LP     Install     (U) 27-APR-2011
HP I64VMS DCPS V2.7                  Full LP     Install     Val 11-APR-2011
HP I64VMS CXX V7.4-4                 Full LP     Install     Val 24-MAR-2011
HP I64VMS C V7.3-18                  Full LP     Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Install     Val 24-MAR-2011
HP I64VMS VMS84I_RMS V2.0            Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_IPC V1.0            Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_UPDATE V5.0         Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_PCSI V2.0           Patch       Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Remove       -  24-MAR-2011
HP I64VMS AVAIL_MAN_BASE V8.4        Full LP     Install     Val 24-MAR-2011
HP I64VMS CDSA V2.4-322              Full LP     Install     Val 24-MAR-2011
HP I64VMS DWMOTIF V1.7               Full LP     Install     Val 24-MAR-2011
HP I64VMS DWMOTIF_SUPPORT V8.4       Full LP     Install     Val 24-MAR-2011
HP I64VMS HPBINARYCHECKER V1.1       Full LP     Install     Val 24-MAR-2011
HP I64VMS KERBEROS V3.1-152          Full LP     Install     Val 24-MAR-2011
HP I64VMS OPENVMS V8.4               Platform    Install     Sys 24-MAR-2011
HP I64VMS SSL V1.4-334               Full LP     Install     Val 24-MAR-2011
HP I64VMS TCPIP V5.7-13              Full LP     Install     Val 24-MAR-2011
HP I64VMS TDC_RT V2.3-20             Full LP     Install     Val 24-MAR-2011
HP I64VMS VMS V8.4                   Oper System Install     Val 24-MAR-2011
HP I64VMS WBEMCIM V2.96-A100211      Full LP     Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Install     Val 24-MAR-2011
------------------------------------ ----------- ----------- --- -----------
59 items found
IA64-2>

Now, which patch do I miss. I can not use JDK 6.0 as JDK 6.0 behaves differently on handling switches like -Dhttp.auth.preference=basic and the like. So all the crashes I explain are basically belonging to java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition
Java HotSpot(TM) Server VM (build 1.5.0-5 07/30/2009-10:16 IA64, mixed mode)
IA64-2>

[Sepp Stadelmann]

In fact svnkit makes use of com.sun.jna and establishes library objects for many purpose. And I wonder why I can not see one for OpenVMS? But regardless of that. If jna is used in certain configurations like MacOS or Linux but not in OpenVMS, why does JSVN run on certtain system sbut not on others?

The issue I have is, why do JVM's at certain sites run a checkout correct and fail after 200 to 300 hunderd file-downloads? Unless JNA dn JVM or JAVA regardless of version 1.5.0-5 or 6.0 is open source, how can I contribute to get the problem solfed?

Hence, I call again for those of you who have a configuration able to checkout i.e. AXIS2 trunks or Maven trunks or just svnkit trunks to tell me which versions of OpenVMS the use and pass to me the output of $ product show history to see if I did not apply all required patches. Particular for CRTLs and the like.

But then ... is such a JVM crash a customer issue or a HP issue?

Sepp

[Sepp Stadelmann]

and slowly I get a problem with this SUN HP relation.

maybe one more fluent with the subject can explain me that,

reading release notes of JDK 1.5.0-5 for OpenVMS one can read

Problems Fixed in JDK 5.0-5

JDK 5.0-5 is based on Sun's J2SE 1.5.0_20 Solaris Reference Release and Olson timezone data file tzdata2009g, and passes all the tests in Sun's Java Compatibility Kit test suite (JCK V1.5a). For additional information on the DST changes, please refer to the Java Technology Software (OpenVMS and Tru64™ UNIX®)website. This JDK is a maintenance release and includes the following fixed problem from HP:

• Previously, incorrect pathname and filename information might have been appended to the name while attempting to rename files. This problem has been fixed.

OK - the HP port of JAVA has passed all Sun Java Compatibility Tests to get the label JAVA

But Sun goes on and delivers something like com.sun.jna.* and there we miss the OpenVMS part.

Either JAVA on OpenVMS can do really without any JNA, or HP should deliver and Sun should include the conformance testing for the OpenVMS JNA part.

One or the other way - this is an incomplete work to who ever it belongs to.

But what is with OpenSource communities delivering OpenSource on Java which makes use of JNA scode and herby becomes platform specific as soon as JNA is introduced, a JNA not supporting OpenVMS?

Sepp

[Volker Halle]

Sepp,

regarding the OpenVMS patches, your system was nearly up-to-date on 24-MAR-2011 (with VMS84I_UPDATE-V0500 applied + IPC V1.0 and RMS V2.0). All the other patch activity - especially on 11-AUG-2011 - was really not necessary (except for LAN-V01, ICAP-V01 and FIBRE_SCSI-V01), as all those patches were already installed or included in VMS84I_UPDATE-V0500. Let's hope that the re-installations of those older patches did not do anything stupid ! OpenVMS patches are still cumulative.

All the symptoms of your JAVA crashes reported so far seem to point to LIB$VM process memory corruptions. Did you note, that the most recent crash contains the hex representation of the string 'SYSTEM...' in the failing VA as opposed to FFFFFFFFFFFFFFF5 before - what did change ?

These type of problems may depend on quota settings and - maybe - incorrect handling of resource problems inside the JAVA code, so they may NOT be easily reproducable except on YOUR system and YOUR user account. As the footprint of these problems, you have the JAVA$JAVA.DMP files, so you need someone, who is willing and capable to look at them, to determine the underlying failure. As the ACCVIOs happen in HP OpenVMS code (LIB$VM) called from HP JAVA code, the chances are high, that HP would be the corrrect target for raising a call. Even if the problem would be caused by some quota/resource problem on your system, ACCVIOs are not the right answer to those problems.

Troubleshooting in this kind of scenario is certainly very complex.

Volker.

[Sepp Stadelmann]

Volker, thank you very much.

my latest topic

JAVA sys vars work with JDK150 but not with JDK60

prevents me to use JDK 6.0 instead of JDK 1.5.0-5 on my new OpenVMS System 8.4.

what I wonder is how a JDK can be downloaded to an OpenVMS 8.3-1H1 or 8.4 where as to my understanding it is liked gaianst some important libraries.

That is why I would like to give JDK 6.0 a try. BUT I am behind a firewall and I don't know how to pass the attributes as given to Java to force authentication on our proxy server. I explained that in my "previous" topic as shown in larger letter above.

BUT when I read about all the problems up to inability to debug with JDK 6.0, I can potentialy raise my hairs ...

I agree with you, OpenVMS patches are cummulative. But as I could not see any difference - I was running jsvn after applying 4 patches and then again after applying the next set of patches - AND it makes no difference at all maybe except the thing you see SYSTEM instead of FFFFFFFFFFFFF5.

There are peoples in OpenVMS engineering. They say it is running at theire site. But so far they ALL lake to tell me about theire UAf quotas, theire system quotas, applied patches, versions of RTL's,  SYSGEN and the like, just all important in such case. And what JVM switches they use to make it run. Also all the required patches to make it run on OpenVMS 8.4 and hopefully 8.3-1H1 too can help, but it seams to be an inability at HP to tell me that.

At the moment I am thinking about how to debugg, run under debugger, but given it crashes in a library of which I do  not have the sources, I feel lost and see this JVM crashes mainly as a HP problem.

Sepp

[Volker Halle]

Sepp,

all those 'libraries' are SHARED libraries, so they are not 'linked into' the JDK .EXE file, but the JDK files are linked 'against them'. So as long as the entrypoints are not changed, which seldom happens and will be done in an upward compatible manner, there is no problem. It's only if the JDK would require some feature or fixes not available in an older version (of OpenVMS or any of the RTLs), that this would be a problem.

I strongly believe, that the problem is NOT in LIBRTL (the LIB$VM routines), but in some other code, which uses them and may corrupt process memory allocated by calls to malloc and such.

There may be some ways to further debug this - just some ideas:

- you could use System Service Logging (SET PROC/SSLOG) to find out, if any system services may fail unexpectedly. Assumption: the JAVA code may not be correctly checking system service return status and therefore causing further problem down the line.

- you could use the debugger and SET BREAK/EXCEPTION to try to catch the ACCVIO (I've never done with with JAVA, so it might not work). Then you could closely look at the failing instruction and find out, where the bad virtual address came from. By looking at the virtual memory around the 'corrupted area', you may get an idea, what's happened

- you could also examine the failing instruction in the JAVA$JAVA.DMP files, try to find the signal array on the stack and the register values at the time of the ACCVIO. Then try to find out, where the bad value got loaded from.

Volker. 

[Sepp Stadelmann]

Volker, I agree 100% with what you say for linking. Yes, it is a symbol vector to define entry points to function and data. As long as interfaces doesn't change as long as one can use JDK 1.5.0-5 or JDK 6.0 ... toward the same shareable images.

Regarding SSLOG ---- I did it with

$ set process/sslog=state=on

$ jsvn checkout <url> trunk7

it runs slowly long AND right after the crash dump file was written I did

$ set process/sslog=state=unload

the SSLOG.DAT was found 1.7 Mio Blocks long and the

$ analyze/sslog runs for ever on a terminal ...

a nogo ... even sending it to an text file and using an editor might become a problem.

using the keep-able debugger to attach to a running process or $ JAVA_DBG can still be used on, that can work. But debugging without any sources .... I feel helpless ... and yes, we use the command DBG> set break/exception all the time when we think our web service crashes intermittent. Lets see after lunch !!!

Sepp

[Volker Halle]

Sepp,

this is like finding a needle in the haystack... Just ignore those 1,699,999 other blocks of the file and find the failing system service ;-) You would need to do some basic testing with this tool first and see, if you can automate checking for failing system services.

Did you try SDA> SHOW PROC and SDA> SHOW PROC/PHD against the JAVA$JAVA.DMP files to look for exhausted quotas ?

Did you check for quota exhaustion in the running process ?

Volker.

[Dennis Handly]

>is such a JVM crash a customer issue or a HP issue?

Do you have any JNIs in your application?  This could make it your issue.

I assume you have a support contract so HP will look at your java problem?

[Ph Vouters]

Sepp and other contributors to this note,

Hang on a bit and I shall install JSVN on an non up to date patched OpenVMS IA64 Version 8.3-1H1 system. Note that I have been unable to build JNA using Java Version 1.5. Hence I will use J2SE Version 6 for OpenVMS servers.

For non aware readers about JNA abilities, refer to http://vouters.dyndns.org/tima/OpenVMS-IA64-Java-JNA-libffi-Porting_JNA_to_OpenVMS_Itanium_servers.html, especially the examples included there. Sepp and other readers will fully aoppreciate my remark about JNA and the multiple possible causes for a process dump when incorrectly coding under an OpenVMS operating system.

However Toine's post is full of promises hence me willing to give JSVN a try. I may end up filling in a special technical article dedicated onto this subject on my Web site. Once and if completed i shall count upon Sepp to both technically and edit review it.

So Sepp and others, best stay tuned.

[Ph Vouters]

Sepp and Toine,

I do need a precision from both of you ! Which JSVN version did you install ? I just dug on Internet and found a JSVN Version 0.8 as well as a jsvn 0.9 dev.jar for download. What should I download to best cope with Sepp's problem and Toine's non problem with JSVN ?

Regards,
Philippe

[Sepp Stadelmann]

Volker,

SDA> show's as you recommend do not show any anomalities as far as I can define it.

The SSLOGdump is simply too big

I redefined the  jsvn command be editing the setup command;

note: jsvn calls now JAVA_DBG.EXE instad of a JAVA.EXE.

This makes it break in the debugger. Where I can do

DBG> set break/exception

DBG> go

DBG> go

But then anomalies over anomalies

1. the jsvn list <url> -R command runs perfect without debugger (that is to say with JAVA.EXE

2. the jsvn list <url> -R command runs not so perfect with the debugger and it outputs into the main terminal window (not debugger)  (see attachment)

Then it runs for a while not shoing any output and it causes exceptions over excpetions (see atachments)

And NO --- I dont think that we hava JNI, but we have a JNA condition. JNA was developed by SUN, and it does not include OpenVMS but many other OS, and I wonder how svnkit can run on other OpenVMS if that should be an issue.

Again: patches is what I need as I can't fix this problem myself unless it is something stupid but then please tell me.

again: I am glad to find a OpenVMS system which claims to run perfect the checkout job. From that system I like to know everything, patches, sysgen params, and uaf quotas etc etc.

Sepp