Operating System - OpenVMS의 주제 Java/JBoss profiling

Java/JBoss profiling

Kristen Haerum — Mon, 25 Jan 2010 10:28:14 GMT

We're having an OpenVMS 8.3 (Itanium) environment running Java 1.5-5 with JBoss 4.2.3GA. (Required VMS patches are installed)

So far (almost) so good: the appserver suffers memory leaks, and must be restarted every few days for the application to be responsive.

Are there any success stories out there using Java profilers on VMS? I've tried a few that behaves well on other platforms, but fails on OpenVMS. E.g. the JBoss profiler claims to be 100% Java, but yields the following output, (indicating a UTF problem?):

UTF ERROR ["JR:[j2se.src.solaris.instrument]EncodingSupport_md.c;1":66]: Failed
to complete iconv_open() setup
%SYSTEM-F-OPCCUS, opcode reserved to customer fault at PC=FFFFF80208AAAB60, PS=0
000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC

DECC$SHR C$SIGNAL gsignal 27939 0000000000001180 FFFFF80208AAAB60
DECC$SHR C$ABORT abort 2831 0000000000000030 FFFFF8020865DA10
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD utfError
5617 00000000000000A0 0000000001F78830
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD utfInitialize
5651 0000000000000250 0000000001F789E0
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD convertUft8ToPlatformString
5744 0000000000000720 0000000001F78EB0
JAVA$INSTRUMENT_SHR INVOCATIONADAPTER appendBootClassPath
18348 0000000000002060 0000000001F7C180
JAVA$INSTRUMENT_SHR INVOCATIONADAPTER Agent_OnLoad
17961 00000000000007D0 0000000001F7A8F0
JAVA$HOTSPOT_SHR THREAD create_vm_init_agents
152702 000000000001B9B0 000000000176BBB0
JAVA$HOTSPOT_SHR THREAD create_vm 152132 000000000001AD80 000000000176AF80
JAVA$HOTSPOT_SHR JNI JNI_CreateJavaVM
146825 0000000000090620 0000000000B271E0
JAVA$JAVA JAVA Java$main_Jacket 11491 0000000000002910 0000000000032910
JAVA$JAVA MAIN_JACKET main 47903 0000000000000340 0000000000047AF0
JAVA$JAVA MAIN_JACKET __main #109 0000000000000100 00000000000478B0
PTHREAD$RTL THD_THREAD thdBase 243309 0000000000005BF0 FFFFF80208541440
PTHREAD$RTL THD_INIT pthread_main 243137 00000000000006C0 FFFFF802084F86C0
0 FFFFFFFF80B26A20 FFFFFFFF80B26A20
DCL 0 000000000006B9C0 000000007AE1B9C0
%TRACE-I-LINENUMBER, Leading '#' specifies a source file record number.
%TRACE-I-END, end of TRACE stack dump

Re: Java/JBoss profiling

Hoff — Mon, 25 Jan 2010 13:24:01 GMT

Start debugging that C code that's part of your 100% Java application; parts of that stuff are clearly written in C.

As you've probably discovered, the routine seeks to convert UTF-8 text to ASCII text by way of Unicode, and there's a bug in there. Or there's a corruption that's slamming that conversion.

My first stop would be process quotas, but if there are leaks around, then the code has severe stability issues that can cause collateral damage all over the place; opens can fail, for instance, secondary to the leak. My second stop would be V8.3-1H1, which is a release HP clearly considers appropriate for all of the Integrity boxes.

In the shortest term, I'd look to restart the stuff daily, and start working with HP support (they've had knowledge of and access to various "private" and "engineering" patches) and/or with upgrades to some or all of the whole stack here.

Stephen Hoffman
HoffmanLabs LLC

Re: Java/JBoss profiling

Kristen Haerum — Mon, 25 Jan 2010 14:06:25 GMT

Thanks to Hoff for your reply.

Concerning the error message -- I agree that it is definitely a C-function. But, it must be a C-function being part of the SDK/JRE layered product. It is NOT part of the profiler, which comes as a collection of JAR files only.

Apparently, EncodingSupport_md.c, is reminiscent from Solaris despite the product is intended for OpenVMS. (!)

Anyway, involving HP support engineers is probably the way to go, although a bit heavy though.

Re: Java/JBoss profiling

H.Becker — Mon, 25 Jan 2010 17:00:02 GMT

>>> ["JR:[j2se.src.solaris.instrument]EncodingSupport_md.c;1":66] <<<

Apparently the Solaris code was compiled on a VMS system (from an ODS5 disk). That doesn't say where the problem is. It still can be a resource problem, which is more likely than a bug in the Solaris code. The fact that this "pure" Java code runs with different run-time libraries for different platforms makes it less likely a bug in the Solaris code.

I would check the known resource limits for Java. Running JAVA$CHECK_ENVIRONMENT.COM may not be enough, but looking into that file gives you some hints which limits/quotas you want to look at.

Re: Java/JBoss profiling

Guy Peleg — Tue, 26 Jan 2010 07:11:37 GMT

Kristen,

I use JRAT to profile Java code. It works
on all platforms including VMS.

http://jrat.sourceforge.net

Very powerful tool.

Regards,

Guy Peleg
Maklee Engineering
www.maklee.com

Re: Java/JBoss profiling

JZ2 — Mon, 01 Feb 2010 13:43:00 GMT

The Agent_OnLoad function is an entry point in a JVMTI native agent. So, I would conclude that the JBoss Profiler is a JVMTI agent.

If you are having java heap problems, have you looked at a heap dump to see what objects are causing the problem? You can generate a heap dump with various options, such as -XX:+HeapDump or -XX:+HeapDumpOnCtrlBreak. See the user guide for more information. You can also use -agentlib:hprof=heap=dump, but the other methods are preferable.

Also, you can generate a GC log using -Xverbosegc, which will give you a lot of information on how the garbage collector is performaing.

Both the GC log and the heap dump can be analyzed using HPjmeter. This is a free tool that you can download from http://www.hp.com/go/hpjmeter.

Re: Java/JBoss profiling

Kristen Haerum — Wed, 10 Feb 2010 10:22:50 GMT

Thanks for contributions; the problem has now been solved now by HP support engineers:

This is a limitation encountered in the DECCRTL library that can be solved either with a logical name, or in JAVA as ASCII is a part of ISO8859-1.

The workaround is as follows:
Install the product VMSI18N from the binary distribution disk:

define this logical in the jboss startup script:
$ def sys$lang EN_US_ISO8859-1

et voila it works !

Btw. this is a problem in Java 1.5 only, according to HP they've solved it in java 1.6.

Re: Java/JBoss profiling

WW304289 — Wed, 10 Feb 2010 16:09:06 GMT

> This is a limitation encountered in the DECCRTL library that can be solved either with a logical name, or in JAVA as ASCII is a part of ISO8859-1.

It is hard to tell without knowing more details about this "limitation in the DECCRTL library", but given the workaround of setting the en_US.iso8859-1 locale, the problem may, actually, be a Java code making unwarranted assumption(s) about built-in C locale.

What exactly this "limitation" is?

-Boris

Re: Java/JBoss profiling

JZ2 — Wed, 10 Feb 2010 17:02:47 GMT

It is good that you were able to get things working. I am still wondering if the jboss profiler is the right tool for the job. If you are experiencing heap issues, then using a standard heap dump, and the GC log frmo -Xverbosegc, together with HPjmeter, might be an easier solution.

I am biased, since I work on HPjmeter.

Re: Java/JBoss profiling

Kristen Haerum — Thu, 11 Feb 2010 07:45:15 GMT

Yes on HP-UX there are more choices; their OpenVMS platform is somewhat odd, and we're lucky to have a few.
Frankly, the JBoss-profiler has not impressed me yet; I'm still try to sharpen trace filters, reducing giant time consuming captures overhead. Also it's snapshots is fuzzy as to what interval and information it really captures.
Dude, it's free software so I shouldn't complain; I'll just have to play around with it, but I know there are better tools out there.

Re: Java/JBoss profiling

Kristen Haerum — Thu, 11 Feb 2010 07:58:56 GMT

Hi Boris

These are fragments of the correspondence with HP:

The problem is Java is asking the CRTL to iconv_open(â ASCIIâ ,â UTF-8â ) and iconv_open(â UTF-8â , â ASCIIâ )

It appears they are not supported on a standard openvms system.

First it requests for ASCII , but on OpenVMS that Character set is really 8859-1.

If the logical is defined, the java application should work because it is going to get ISO8859-1 as the default language.

Actually we should call decc$iconv_open(â ISO8859-1â , â UTF-8â ) and decc$iconv_open(â UTF-8â , â ISO8859-1â ),

But both should work.

Since Java gets the default lang first, the JBOSS debugger shouldnâ t fail if sys$lang EN_US_ISO8859-1

-Kristen

Re: Java/JBoss profiling

WW304289 — Thu, 11 Feb 2010 16:13:06 GMT

Hi Kristen,

thanks for the reply.

The names of codesets are implementation-defined, including the codeset associated with program's default locale. Attached shapshot illustrates it. The thing is that Linux (and, probably, other platforms) provides codeset converters for codeset associated with program's default locale while VMS does not.

There is no standard requirement I know of to provide codeset converters for any particular codeset, including the default codeset, but not providing converters for default codeset looks like a deficiency to me. It is, actually, easy to fix: create codeset convertes for ASCII as symbolic links to codeset converters for ISO8859-1:

ASCII_UCS-2.ICONV -> ISO8859-1_UCS-2.ICONV
ASCII_UCS-4.ICONV -> ISO8859-1_UCS-4.ICONV
ASCII_UTF-8.ICONV -> ISO8859-1_UTF-8.ICONV
UCS-2_ISO8859-1.ICONV -> UCS-2_ASCII.ICONV
...

> Actually we should call decc$iconv_open(Ã¢Â Â ISO8859-1Ã¢Â Â , Ã¢Â Â UTF-8Ã¢Â Â ) and decc$iconv_open(Ã¢Â Â UTF-8Ã¢Â Â , Ã¢Â Â ISO8859-1Ã¢Â Â ),

Yes, I think this is the right fix, instead of defining sys$lang logical. Except decc$ prefix is unnecessary.

Thanks again,
-Boris

x.c
---
#include
#include
#include

int main() {
setlocale(LC_ALL, "");
puts(nl_langinfo(CODESET));
}

VMS
---
$ run x.exe
ASCII
$ define sys$lang EN_US_ISO8859-1
$ run x.exe
ISO8859-1
$

$ dir SYS$I18N_ICONV:*ASCII*.ICONV
%DIRECT-W-NOFILES, no files found
$

Linux
-----
$ x.out
ANSI_X3.4-1968
$ export LANG=en_US
$ x.out
ISO-8859-1
$

$ iconv --list | grep ANSI_X3.4-1968
ANSI_X3.4-1968//
$

Re: Java/JBoss profiling

Cass Witkowski — Mon, 13 Sep 2010 22:30:29 GMT

What JBOSS are you using?

Thanks,

Cass

Re: Java/JBoss profiling

Ph Vouters — Tue, 19 Jul 2011 22:04:59 GMT

Hi,

My article I recently wrote after discovering the original post in Google's comp.os.vms news group is coming late. Anyhow here it is : http://vouters.dyndns.org/tima/OpenVMS-Java-JBoss-JBoss_Profiler_2_causing_a_JVM_crash_dump.html

If this can also help someone or leadstoward the solution I will be glad.

Philippe