JDK 1.5.0 & JDK 6.0 JVM crash executing JSVN, a Java Apps

[Sepp Stadelmann]

Hi

I have applied some more patches to the system, but no big break through ...

So I retried again my jsvn checkout command and in the attachment I can show you some more insigth.

And I fully agree with you;

The onse with the LIBRTL source code should look at it !!! ----> HP OpenVMS Engineering

Thank you very much to help me speeding up this process.

Sepp

[Ph Vouters]

Would running Java under the debugger control help diagnose the real problem ? For more information about running Java under the debugger control, refer to http://vouters.dyndns.org/tima/OpenVMS-Java-V6.0-A_user_experience_with_OpenVMS_IA64_Java_V6.0.html (see temporary workaround for problem one).

I seem to recollect JSVN is dependant upon JNA.  Do you confirm this ? If yes, there are lots of causes for a JVM crash !

If this can help I should be glad.

Philippe

[Toine_1]

Hi,

Did you set the logical names below before you run jsvn?

$!
$ define/job/nolog decc$argv_parse_style enable
$ define/job/nolog decc$efs_case_preserve enable

And make sure the logical name DECC$EFS_CASE_SPECIAL is not defined.

$!
$ if f$edit(f$trnlnm("DECC$EFS_CASE_SPECIAL"),"UPCASE") .eqs. "ENABLE"
$ then
$    write sys$output " "
$    write sys$output "JSVNSETUP: DECC$EFS_CASE_SPECIAL defined! Please deassign logical otherwise jsvn might not function properly!
$    write sys$output " "
$ endif

We are using JSVN on OpenVMS 8.3-1H1 and 8.4 and it works fine with Java 1.5 and 1.6.

Regards,

/Toine

[Sepp Stadelmann]

this are my DECC logicals AS SET AFTER A CRASH

A    trunk6/CHANGES.txt
A    trunk6/svnkit-dav
OpenVMS stack trace:

IA64-2>show  log decc*

(LNM$PROCESS_TABLE)

  "DECC$EFS_CHARSET" = "true"
  "DECC$FD_LOCKING" = "1"

(LNM$JOB_B16ECB00)

  "DECC$ARGV_PARSE_STYLE" = "enable"
  "DECC$EFS_CASE_PRESERVE" = "enable"

(SERVER_24CCCC81)

(LNM$GROUP_000201)

(LNM$SYSTEM_TABLE)

  "DECC$CRTLMAP" = "SYS$SHARE:DECC$SHR"
  "DECC$SHR_AV" = "DECC$SHR"

(LNM$SYSCLUSTER_TABLE)

(DECW$LOGICAL_NAMES)
IA64-2>

And now I would like to know form thouse of you which say that JSVN runs if JSVN and the given version of OpenVMS is able to checkout a very large source repository at once.

I would like to know in this case the exact patch leven and the version of JAVA / JVM they use.

please do me a favor and deposit the installed products here. my onces are.

IA64-2>product show hist
------------------------------------ ----------- ----------- --- -----------
PRODUCT                              KIT TYPE    OPERATION   VAL DATE
------------------------------------ ----------- ----------- --- -----------
HP I64VMS TCPIP_NTP_PAT V5.7-ECO1B   Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_DEBUG V1.0          Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_FIBRE_SCSI V1.0     Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_ICAP V1.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_LAN V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_IPC V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_MUP V2.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_MUP V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_PCSI V2.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_PCSI V1.0           Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_RMS V2.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_RMS V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_SYS V1.0            Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V3.0         Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V2.0         Patch       Install     Val 11-AUG-2011
HP I64VMS VMS84I_UPDATE V1.0         Patch       Install     Val 11-AUG-2011
HP I64VMS GNV V2.1-3                 Full LP     Install     Val 10-AUG-2011
HP I64VMS GNV V2.1-3                 Full LP     Remove       -  10-AUG-2011
HP I64VMS VMS84I_DEBUG V1.0          Patch       Install     Val 08-AUG-2011
HP I64VMS C V7.3-19                  Full LP     Install     Val 08-AUG-2011
HP I64VMS C V7.3-18                  Full LP     Remove       -  08-AUG-2011
HP I64VMS VMS84I_LAN V1.0            Patch       Install     Val 08-AUG-2011
HP I64VMS LTT V4.13-0                Full LP     Install     Val 03-AUG-2011
HP I64VMS VMS84I_FIBRE_SCSI V1.0     Patch       Install     Val 22-JUL-2011
HP I64VMS LTT V4.13-0                Full LP     Install     Val 14-JUL-2011
HP I64VMS IDESERVER V6.52            Full LP     Install     Val 12-JUL-2011
HP I64VMS CSWS_PHP V2.20             Full LP     Install     Val 06-JUL-2011
HP I64VMS CSWS_PERL V2.1             Full LP     Install     (U) 06-JUL-2011
HP IA64VMS WSIT V3.0                 Full LP     Install     (U) 06-JUL-2011
HP I64VMS GNV V2.1-3                 Full LP     Install     Val 05-JUL-2011
HP I64VMS CSWS_JAVA V3.1             Full LP     Install     Val 05-JUL-2011
HP I64VMS CSWS V2.2                  Full LP     Install     Val 05-JUL-2011
HP I64VMS JAVA150 V1.5-5             Full LP     Install     Val 05-JUL-2011
HP I64VMS JAVA60 V1.6-2_P1           Full LP     Install     Val 05-JUL-2011
HP I64VMS PERL586_UPDATE V2.0        Patch       Install     Val 09-MAY-2011
HP I64VMS PERL V5.8-6                Full LP     Install     (U) 09-MAY-2011
IBM I64VMS WMQCLIENT V6.0            Full LP     Install     (U) 27-APR-2011
HP I64VMS DCPS V2.7                  Full LP     Install     Val 11-APR-2011
HP I64VMS CXX V7.4-4                 Full LP     Install     Val 24-MAR-2011
HP I64VMS C V7.3-18                  Full LP     Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Install     Val 24-MAR-2011
HP I64VMS VMS84I_RMS V2.0            Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_IPC V1.0            Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_UPDATE V5.0         Patch       Install     Val 24-MAR-2011
HP I64VMS VMS84I_PCSI V2.0           Patch       Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Remove       -  24-MAR-2011
HP I64VMS AVAIL_MAN_BASE V8.4        Full LP     Install     Val 24-MAR-2011
HP I64VMS CDSA V2.4-322              Full LP     Install     Val 24-MAR-2011
HP I64VMS DWMOTIF V1.7               Full LP     Install     Val 24-MAR-2011
HP I64VMS DWMOTIF_SUPPORT V8.4       Full LP     Install     Val 24-MAR-2011
HP I64VMS HPBINARYCHECKER V1.1       Full LP     Install     Val 24-MAR-2011
HP I64VMS KERBEROS V3.1-152          Full LP     Install     Val 24-MAR-2011
HP I64VMS OPENVMS V8.4               Platform    Install     Sys 24-MAR-2011
HP I64VMS SSL V1.4-334               Full LP     Install     Val 24-MAR-2011
HP I64VMS TCPIP V5.7-13              Full LP     Install     Val 24-MAR-2011
HP I64VMS TDC_RT V2.3-20             Full LP     Install     Val 24-MAR-2011
HP I64VMS VMS V8.4                   Oper System Install     Val 24-MAR-2011
HP I64VMS WBEMCIM V2.96-A100211      Full LP     Install     Val 24-MAR-2011
HP I64VMS WBEMPROVIDERS V2.0-4       Full LP     Install     Val 24-MAR-2011
------------------------------------ ----------- ----------- --- -----------
59 items found
IA64-2>

Now, which patch do I miss. I can not use JDK 6.0 as JDK 6.0 behaves differently on handling switches like -Dhttp.auth.preference=basic and the like. So all the crashes I explain are basically belonging to java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition
Java HotSpot(TM) Server VM (build 1.5.0-5 07/30/2009-10:16 IA64, mixed mode)
IA64-2>

[Sepp Stadelmann]

In fact svnkit makes use of com.sun.jna and establishes library objects for many purpose. And I wonder why I can not see one for OpenVMS? But regardless of that. If jna is used in certain configurations like MacOS or Linux but not in OpenVMS, why does JSVN run on certtain system sbut not on others?

The issue I have is, why do JVM's at certain sites run a checkout correct and fail after 200 to 300 hunderd file-downloads? Unless JNA dn JVM or JAVA regardless of version 1.5.0-5 or 6.0 is open source, how can I contribute to get the problem solfed?

Hence, I call again for those of you who have a configuration able to checkout i.e. AXIS2 trunks or Maven trunks or just svnkit trunks to tell me which versions of OpenVMS the use and pass to me the output of $ product show history to see if I did not apply all required patches. Particular for CRTLs and the like.

But then ... is such a JVM crash a customer issue or a HP issue?

Sepp

[Sepp Stadelmann]

and slowly I get a problem with this SUN HP relation.

maybe one more fluent with the subject can explain me that,

reading release notes of JDK 1.5.0-5 for OpenVMS one can read

Problems Fixed in JDK 5.0-5

JDK 5.0-5 is based on Sun's J2SE 1.5.0_20 Solaris Reference Release and Olson timezone data file tzdata2009g, and passes all the tests in Sun's Java Compatibility Kit test suite (JCK V1.5a). For additional information on the DST changes, please refer to the Java Technology Software (OpenVMS and Tru64™ UNIX®)website. This JDK is a maintenance release and includes the following fixed problem from HP:

• Previously, incorrect pathname and filename information might have been appended to the name while attempting to rename files. This problem has been fixed.

OK - the HP port of JAVA has passed all Sun Java Compatibility Tests to get the label JAVA

But Sun goes on and delivers something like com.sun.jna.* and there we miss the OpenVMS part.

Either JAVA on OpenVMS can do really without any JNA, or HP should deliver and Sun should include the conformance testing for the OpenVMS JNA part.

One or the other way - this is an incomplete work to who ever it belongs to.

But what is with OpenSource communities delivering OpenSource on Java which makes use of JNA scode and herby becomes platform specific as soon as JNA is introduced, a JNA not supporting OpenVMS?

Sepp

[Volker Halle]

Sepp,

regarding the OpenVMS patches, your system was nearly up-to-date on 24-MAR-2011 (with VMS84I_UPDATE-V0500 applied + IPC V1.0 and RMS V2.0). All the other patch activity - especially on 11-AUG-2011 - was really not necessary (except for LAN-V01, ICAP-V01 and FIBRE_SCSI-V01), as all those patches were already installed or included in VMS84I_UPDATE-V0500. Let's hope that the re-installations of those older patches did not do anything stupid ! OpenVMS patches are still cumulative.

All the symptoms of your JAVA crashes reported so far seem to point to LIB$VM process memory corruptions. Did you note, that the most recent crash contains the hex representation of the string 'SYSTEM...' in the failing VA as opposed to FFFFFFFFFFFFFFF5 before - what did change ?

These type of problems may depend on quota settings and - maybe - incorrect handling of resource problems inside the JAVA code, so they may NOT be easily reproducable except on YOUR system and YOUR user account. As the footprint of these problems, you have the JAVA$JAVA.DMP files, so you need someone, who is willing and capable to look at them, to determine the underlying failure. As the ACCVIOs happen in HP OpenVMS code (LIB$VM) called from HP JAVA code, the chances are high, that HP would be the corrrect target for raising a call. Even if the problem would be caused by some quota/resource problem on your system, ACCVIOs are not the right answer to those problems.

Troubleshooting in this kind of scenario is certainly very complex.

Volker.

[Sepp Stadelmann]

Volker, thank you very much.

my latest topic

JAVA sys vars work with JDK150 but not with JDK60

prevents me to use JDK 6.0 instead of JDK 1.5.0-5 on my new OpenVMS System 8.4.

what I wonder is how a JDK can be downloaded to an OpenVMS 8.3-1H1 or 8.4 where as to my understanding it is liked gaianst some important libraries.

That is why I would like to give JDK 6.0 a try. BUT I am behind a firewall and I don't know how to pass the attributes as given to Java to force authentication on our proxy server. I explained that in my "previous" topic as shown in larger letter above.

BUT when I read about all the problems up to inability to debug with JDK 6.0, I can potentialy raise my hairs ...

I agree with you, OpenVMS patches are cummulative. But as I could not see any difference - I was running jsvn after applying 4 patches and then again after applying the next set of patches - AND it makes no difference at all maybe except the thing you see SYSTEM instead of FFFFFFFFFFFFF5.

There are peoples in OpenVMS engineering. They say it is running at theire site. But so far they ALL lake to tell me about theire UAf quotas, theire system quotas, applied patches, versions of RTL's,  SYSGEN and the like, just all important in such case. And what JVM switches they use to make it run. Also all the required patches to make it run on OpenVMS 8.4 and hopefully 8.3-1H1 too can help, but it seams to be an inability at HP to tell me that.

At the moment I am thinking about how to debugg, run under debugger, but given it crashes in a library of which I do  not have the sources, I feel lost and see this JVM crashes mainly as a HP problem.

Sepp

[Volker Halle]

Sepp,

all those 'libraries' are SHARED libraries, so they are not 'linked into' the JDK .EXE file, but the JDK files are linked 'against them'. So as long as the entrypoints are not changed, which seldom happens and will be done in an upward compatible manner, there is no problem. It's only if the JDK would require some feature or fixes not available in an older version (of OpenVMS or any of the RTLs), that this would be a problem.

I strongly believe, that the problem is NOT in LIBRTL (the LIB$VM routines), but in some other code, which uses them and may corrupt process memory allocated by calls to malloc and such.

There may be some ways to further debug this - just some ideas:

- you could use System Service Logging (SET PROC/SSLOG) to find out, if any system services may fail unexpectedly. Assumption: the JAVA code may not be correctly checking system service return status and therefore causing further problem down the line.

- you could use the debugger and SET BREAK/EXCEPTION to try to catch the ACCVIO (I've never done with with JAVA, so it might not work). Then you could closely look at the failing instruction and find out, where the bad virtual address came from. By looking at the virtual memory around the 'corrupted area', you may get an idea, what's happened

- you could also examine the failing instruction in the JAVA$JAVA.DMP files, try to find the signal array on the stack and the register values at the time of the ACCVIO. Then try to find out, where the bad value got loaded from.

Volker. 

[Sepp Stadelmann]

Volker, I agree 100% with what you say for linking. Yes, it is a symbol vector to define entry points to function and data. As long as interfaces doesn't change as long as one can use JDK 1.5.0-5 or JDK 6.0 ... toward the same shareable images.

Regarding SSLOG ---- I did it with

$ set process/sslog=state=on

$ jsvn checkout <url> trunk7

it runs slowly long AND right after the crash dump file was written I did

$ set process/sslog=state=unload

the SSLOG.DAT was found 1.7 Mio Blocks long and the

$ analyze/sslog runs for ever on a terminal ...

a nogo ... even sending it to an text file and using an editor might become a problem.

using the keep-able debugger to attach to a running process or $ JAVA_DBG can still be used on, that can work. But debugging without any sources .... I feel helpless ... and yes, we use the command DBG> set break/exception all the time when we think our web service crashes intermittent. Lets see after lunch !!!

Sepp