Problem creating Java Virtual Machine

[Yyrkoon_]

Hi all,

I am having a lot of problems creating JVM, I work with 4 different Alpha (DS10 617MHz) clusters, all them with the same configuration, but different processes running. When I try to run a Java program on them (same Java program in all servers), two of the clusters run perfectly, but other two "Cannot create Java Virtual Machine", it does not matter what I try, I have tried commands like -"Xss64k" -"Xms4m" -"Xmx4m" -client and so on. The only working option is reboot the cluster (even running autogen) and after that Java is able to create the virtual machine during a couple of days, then the problem starts again.

I am getting crazy. Any awesome idea from you?

Thanks in advance.

MIN_NPAGEDYN    = 1998848

MIN_NPAGEDYN    = 670000

MIN_PAGEDYN     = 289000

SWAPFILE        = 0             ! don’t re-size the SWAPFILE on AUTOGEN runs 

MIN_gblsections = 750           ! required for DECwindows MOTIF  

MIN_GBLPAGES    = 278528        ! # free (unused) pages req.

MIN_GBLPAGFIL   = 6346          !   free (unused) space req. 

MIN_GBLSECTIONS = 1654          ! # free (unused) sects. req.

[abrsvc]

MOre info needed:

OpenVMS version

Software products versions (Java being one)

Any sysgen parameter differences between the machiens (other than node names and scs id numbers)?

Any specific processes running on teh machines that fail over time?

Pagefile sizes different on the machines in question?

[H.Becker]

>>>MOre info needed:

True.

... and you may want to check your environment with one of the command procedures found in

$ dir 'f$parse(f$log("JAVA$JRE_HOME_VMS")-"]"+".-.com]",,"*.com",)

 or use the wizard  from that directory to set up  some debug logicals for java.

And, as usual, what was the java command and what was/is the output/message in the working and failing environment?

Finally, when it fails I would start with simple things like

$ java -version

 to narrow down the problem.

[Hoff]

---

@Yyrkoon_ wrote:

The only working option is reboot the cluster (even running autogen) and after that Java is able to create the virtual machine during a couple of days, then the problem starts again.

---

That a system or cluster reboot clears the problem is usually indicative of a leak of and the exhaustion of some system-wide shared resource, or potentially of some application-specific shared resource. 

Pagefile, shared logical name tables, non-paged pool, locks, etc.   Maybe even disk space, particularly if there's a clean-up performed in the local system startups.

MIN settings for AUTOGEN are just that, minimums.  Actual reality can require different (and potentially larger) settings, though which settings are involved here is not at all clear.

Usual recommendations: review and apply patches.   Size your dump file, though AUTOGEN has done that for you if there's no DUMPFILE = 0 or DUMPFILE = too-small-value lurking in your MODPARAMS.  

Then get somebody to look at the system and the cluster when it next acts up.  If this is a production system and you can't delay rebooting for that access and that system review and that troubleshooting, then use the console to force crashdumps to be written when you reboot the cluster.  You'll need to ensure that your system dump files are present on each system (not shared) and are sufficiently sized for the crashes based on the amount of memory configured.  

Slightly more effort: you can load and configure T4, or can use the recording capabilities of MONITOR and some local DCL procedures, to collect system performance data, and that can help spot resource usage trends.

If you want to discuss this here, there's going to be a whole lot more information necessary, starting with the SDA> CLUE CONFIG, and it's going to be some effort to post data and then dig through and spot any resource usage trends, or a low-level configuration error, via this forum software.

[Yyrkoon_]

Thanks for your quick response, you can find attached some of the requested data.

I agree with Hoff that this seems as a memory leak, I have looked for a command to clean the heap, but I have not been able to find it (if it exists). Anyway I will add the dump value to modparams for the next reboot as we don't have this set.

I hope you can find something wrong on any of my servers parameters.

Once again... many thanks for your help.

OpenVMS V8.3

ASTLM is 4096 -- recommended minumum is: 300

BIOLM is 150 -- recommended minumum is: 150

BYTLM is 400256 -- recommended minumum is: 400000

DIOLM is 400 -- recommended minumum is: 150

ENQLM is 5000 -- recommended minumum is: 4000

FILLM is 4096 -- recommended minumum is: 4096

PGFLQUOTA is 2098000 -- recommended minumum is: 2097152

PRCLM is 60 -- recommended minumum is: 10

TQELM is 500 -- recommended minumum is: 100

WSQUOTA is 8192 -- recommended minumum is: 4096

CHANNELCNT is 4096 -- recommended minumum is: 4096

java version "1.5.0"

Java(TM) 2 Runtime Environment, Standard Edition

Fast VM (build 1.5.0-8, build J2SDK.v.1.5.0:02/08/2012-17:12, native threads, jit_150)

Sometimes JVM is not able to run even “java –version”, take into account that java is currently working…

MODPARAMS FILE

!

SCSSYSTEMID=17798

SCSNODE="XXXXXXXX "

VAXCLUSTER=2

EXPECTED_VOTES=3

VOTES=1

DISK_QUORUM="$1$DKB200     "

QDSKVOTES=1

QDSKINTERVAL=3

ALLOCLASS=1

LOCKDIRWT=0

NISCS_CONV_BOOT=0

NISCS_LOAD_PEA0=1

MSCP_LOAD=1

MSCP_SERVE_ALL=1

RECNXINTERVAL=120

NISCS_PORT_SERV=0

!

!****************************************************************************

! This section contains any parameters found in

! SYS$SYSDEVICE:[SYS0.SYSEXE]MODPARAMS.DAT

!

!vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv

!

! CLUSTER_CONFIG appending for ADD operation on 9-JUL-2002 14:14:42.06

AGEN$INCLUDE_PARAMS SYS$MANAGER:AGEN$NEW_NODE_DEFAULTS.DAT

INTERCONNECT="NI"

BOOTNODE="NO"

! CLUSTER_CONFIG end

!

! Set by Factory Installed Software on 9-JUL-2002

MIN_KSTACKPAGES=3

WINDOW_SYSTEM = 1

! MIN_NPAGEDYN = 1998848

! End of Factory Installed Software settings

!

!*** The following entries have been added by the RMS remedial kit

!*

ADD_PIOPAGES = 100     !***Entered by RMS (ASB increase)

ADD_IMGIOCNT = 100     !***Entered by RMS (ASB increase)

MIN_PIOPAGES = 675     !***Entered by RMS (ASB increase)

MIN_IMGIOCNT = 228     !***Entered by RMS (ASB increase)

!*

!*** End of RMS remedial kit entries

!

shadowing = 2

shadow_sys_disk = 1

!

!PB 28/10/2006 Requirements for MQ-series

min_gblsections = 2500

min_gblpages = 1000000

min_gblpagfil = 1000000

!min_channelcnt = 2500

!

! Requirement for JAVA -

!

min_channelcnt = 4096

!

! Solution for insufficient nonpaged pool -

!

min_npagedyn = 32604160

! min_npagevir = 57057280

!

!^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[Hoff]

It's not at all clear MODPARAMS.DAT and the other settings are the source or even have any particular relation to what's going on here, and a single snapshot of the system environment will seldom capture and identify a quota leak as that's inherently a trend over time.

Which means you're faced with the choice of learning rather more how VMS and Java works and how to troubleshoot the misbehavior and with using previously-mentioned tools to monitor the usage trends — that's usually more than can be described in forum postings, and requires rather more time expended to gain the associated experience and to make the required mistakes — or get somebody in that does understand how to troubleshoot this and similar cases, or do what many folks will do when presented with this behavior, and just reboot this box periodically and ignore the problem.  

Do review the Java installation guide for its recommended settings and confirm that that those (usually minimums) are are in use, and also confirm that the required OpenVMS patches are all installed, too.  It's also possible that these "identical" AlphaServer boxes are just under-configured, and some other activity on the system is pushing Java over into failure.

[H.Becker]

As someone else usually points out, actual commands and actual output would help.

This includes the JAVA symbol, usually set up by JAVA$150_SETUP.COM, the JAVA$ logicals and the symbol CLASSPATH. And then there are the (in)famous DECC$ feature logicals which influence the behavior of Java as well.

That said, as far as I understand the real error message you see is "Could not create the Java virtual machine."

On VMS the JVM is determined by the logical name JAVA$JVM_SHR. From the shown version output in your environment it seems to point to JAVA$FVM_SHR.EXE, the fast VM.

Again, I suggest to set some Java debug logicals:

$ def _JAVA_LAUNCHER_DEBUG  1
$ def JAVA$PRINT_COMMAND_ARGS TRUE
$ def JAVA_PLUGIN_TRACE TRUE
$ def JAVA$EXEC_TRACE TRUE

With that you should see something like:

$ java -version -verbose:class
0: semmel$dka400:[sysa.syscommon.][java$150.bin]java$java.exe;1 
1: -version 
2: -verbose:class 
----_JAVA_LAUNCHER_DEBUG----
JRE path is /sys$common/java$150/jre
jvm.cfg[0] = ->-fast<-
jvm.cfg[1] = ->-fast32<-
jvm.cfg[2] = ->-fast64<-
jvm.cfg[3] = ->-classic<-
jvm.cfg[4] = ->-native<-
jvm.cfg[5] = ->-green<-
1 micro seconds to parse jvm.cfg
Default VM: fast
Does `/sys$common/java$150/jre/lib/alpha/fastvm/java$fvm_shr.exe' exist ... yes.
JVM path is /sys$common/java$150/jre/lib/alpha/fastvm/java$fvm_shr.exe
1 micro seconds to LoadJavaVM
JavaVM args:
    version 0x00010002, ignoreUnrecognized is JNI_FALSE, nOptions is 2
    option[ 0] = '-Djava.class.path=.'
    option[ 1] = '-Dsun.java.launcher=SUN_STANDARD'
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition
Fast VM (build 1.5.0-8, build J2SDK.v.1.5.0:02/08/2012-17:12, native threads, jit_150)
$ 

Running "java -version" doesn't really do anything with the VM: there are no classes loaded, as far as one can trust the -verbose:class output. On the other hand, the symbol JAVA usually is "$ SYS$COMMON:[JAVA$150.bin]java$java
" and that image is linked with JAVA$JVM_SHR, here JAVA$FVM_SHR.EXE - as analyze/image or any other suitable tool will show you:

$ shiml -u semmel$dka400:[sysa.syscommon.][java$150.bin]java$java.exe;1 
SHareable IMage dependency List (Alpha), version 1.5
 [ -> translated logical name ] 
 [ (self) - self reference ]

JAVA$JAVA_VMS_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha]JAVA$JAVA_VMS_SHR.EXE;
CMA$TIS_SHR (self)
LIBRTL
SYS$PUBLIC_VECTORS
LIBOTS
DECC$SHR -> SYS$SHARE:DECC$SHR_EV56
DPML$SHR (self)
PTHREAD$RTL (self)
SYS$BASE_IMAGE
JAVA$JVM_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha.FASTVM]JAVA$FVM_SHR.EXE;
JAVA$JAVA_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha]JAVA$JAVA_SHR.EXE;
JAVA$VERIFY_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha]JAVA$VERIFY_SHR.EXE;
JAVA$HPI_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha.NATIVE_THREADS]JAVA$HPI_SHR.EXE;
JAVA$ZIP_SHR -> SYS$COMMON:[JAVA$150.JRE.LIB.Alpha]JAVA$ZIP_SHR.EXE;
$ 

The image activator would complain if it couldn't find or activate the VM image. However, from the above java debug output one might interpret that java.exe does a lib$fis on that image as well to find some symbols and one may conclude that in your error case something fails here.

I can't reproduce your problem and so I don't know whether the Java debug code will print more information in your error case, but with the debug code you might get a clue what's (not) going on.