Java/JBoss profiling

[Kristen Haerum]

We're having an OpenVMS 8.3 (Itanium) environment running Java 1.5-5 with JBoss 4.2.3GA. (Required VMS patches are installed)

So far (almost) so good: the appserver suffers memory leaks, and must be restarted every few days for the application to be responsive.

Are there any success stories out there using Java profilers on VMS? I've tried a few that behaves well on other platforms, but fails on OpenVMS. E.g. the JBoss profiler claims to be 100% Java, but yields the following output, (indicating a UTF problem?):

UTF ERROR ["JR:[j2se.src.solaris.instrument]EncodingSupport_md.c;1":66]: Failed
to complete iconv_open() setup
%SYSTEM-F-OPCCUS, opcode reserved to customer fault at PC=FFFFF80208AAAB60, PS=0
000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC

DECC$SHR C$SIGNAL gsignal 27939 0000000000001180 FFFFF80208AAAB60
DECC$SHR C$ABORT abort 2831 0000000000000030 FFFFF8020865DA10
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD utfError
5617 00000000000000A0 0000000001F78830
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD utfInitialize
5651 0000000000000250 0000000001F789E0
JAVA$INSTRUMENT_SHR ENCODINGSUPPORT_MD convertUft8ToPlatformString
5744 0000000000000720 0000000001F78EB0
JAVA$INSTRUMENT_SHR INVOCATIONADAPTER appendBootClassPath
18348 0000000000002060 0000000001F7C180
JAVA$INSTRUMENT_SHR INVOCATIONADAPTER Agent_OnLoad
17961 00000000000007D0 0000000001F7A8F0
JAVA$HOTSPOT_SHR THREAD create_vm_init_agents
152702 000000000001B9B0 000000000176BBB0
JAVA$HOTSPOT_SHR THREAD create_vm 152132 000000000001AD80 000000000176AF80
JAVA$HOTSPOT_SHR JNI JNI_CreateJavaVM
146825 0000000000090620 0000000000B271E0
JAVA$JAVA JAVA Java$main_Jacket 11491 0000000000002910 0000000000032910
JAVA$JAVA MAIN_JACKET main 47903 0000000000000340 0000000000047AF0
JAVA$JAVA MAIN_JACKET __main #109 0000000000000100 00000000000478B0
PTHREAD$RTL THD_THREAD thdBase 243309 0000000000005BF0 FFFFF80208541440
PTHREAD$RTL THD_INIT pthread_main 243137 00000000000006C0 FFFFF802084F86C0
0 FFFFFFFF80B26A20 FFFFFFFF80B26A20
DCL 0 000000000006B9C0 000000007AE1B9C0
%TRACE-I-LINENUMBER, Leading '#' specifies a source file record number.
%TRACE-I-END, end of TRACE stack dump

[Hoff]

Start debugging that C code that's part of your 100% Java application; parts of that stuff are clearly written in C.

As you've probably discovered, the routine seeks to convert UTF-8 text to ASCII text by way of Unicode, and there's a bug in there. Or there's a corruption that's slamming that conversion.

My first stop would be process quotas, but if there are leaks around, then the code has severe stability issues that can cause collateral damage all over the place; opens can fail, for instance, secondary to the leak. My second stop would be V8.3-1H1, which is a release HP clearly considers appropriate for all of the Integrity boxes.

In the shortest term, I'd look to restart the stuff daily, and start working with HP support (they've had knowledge of and access to various "private" and "engineering" patches) and/or with upgrades to some or all of the whole stack here.

Stephen Hoffman
HoffmanLabs LLC

[Kristen Haerum]

Thanks to Hoff for your reply.

Concerning the error message -- I agree that it is definitely a C-function. But, it must be a C-function being part of the SDK/JRE layered product. It is NOT part of the profiler, which comes as a collection of JAR files only.

Apparently, EncodingSupport_md.c, is reminiscent from Solaris despite the product is intended for OpenVMS. (!)

Anyway, involving HP support engineers is probably the way to go, although a bit heavy though.

[H.Becker]

>>> ["JR:[j2se.src.solaris.instrument]EncodingSupport_md.c;1":66] <<<

Apparently the Solaris code was compiled on a VMS system (from an ODS5 disk). That doesn't say where the problem is. It still can be a resource problem, which is more likely than a bug in the Solaris code. The fact that this "pure" Java code runs with different run-time libraries for different platforms makes it less likely a bug in the Solaris code.

I would check the known resource limits for Java. Running JAVA$CHECK_ENVIRONMENT.COM may not be enough, but looking into that file gives you some hints which limits/quotas you want to look at.

[Guy Peleg]

Kristen,

I use JRAT to profile Java code. It works
on all platforms including VMS.

http://jrat.sourceforge.net

Very powerful tool.

Regards,

Guy Peleg
Maklee Engineering
www.maklee.com

[JZ2]

The Agent_OnLoad function is an entry point in a JVMTI native agent. So, I would conclude that the JBoss Profiler is a JVMTI agent.

If you are having java heap problems, have you looked at a heap dump to see what objects are causing the problem? You can generate a heap dump with various options, such as -XX:+HeapDump or -XX:+HeapDumpOnCtrlBreak. See the user guide for more information. You can also use -agentlib:hprof=heap=dump, but the other methods are preferable.

Also, you can generate a GC log using -Xverbosegc, which will give you a lot of information on how the garbage collector is performaing.

Both the GC log and the heap dump can be analyzed using HPjmeter. This is a free tool that you can download from http://www.hp.com/go/hpjmeter.

[Kristen Haerum]

Thanks for contributions; the problem has now been solved now by HP support engineers:

This is a limitation encountered in the DECCRTL library that can be solved either with a logical name, or in JAVA as ASCII is a part of ISO8859-1.

The workaround is as follows:
Install the product VMSI18N from the binary distribution disk:

define this logical in the jboss startup script:
$ def sys$lang EN_US_ISO8859-1

et voila it works !

Btw. this is a problem in Java 1.5 only, according to HP they've solved it in java 1.6.

[WW304289]

> This is a limitation encountered in the DECCRTL library that can be solved either with a logical name, or in JAVA as ASCII is a part of ISO8859-1.

It is hard to tell without knowing more details about this "limitation in the DECCRTL library", but given the workaround of setting the en_US.iso8859-1 locale, the problem may, actually, be a Java code making unwarranted assumption(s) about built-in C locale.

What exactly this "limitation" is?

-Boris

[JZ2]

It is good that you were able to get things working. I am still wondering if the jboss profiler is the right tool for the job. If you are experiencing heap issues, then using a standard heap dump, and the GC log frmo -Xverbosegc, together with HPjmeter, might be an easier solution.

I am biased, since I work on HPjmeter.

[Kristen Haerum]

Yes on HP-UX there are more choices; their OpenVMS platform is somewhat odd, and we're lucky to have a few.
Frankly, the JBoss-profiler has not impressed me yet; I'm still try to sharpen trace filters, reducing giant time consuming captures overhead. Also it's snapshots is fuzzy as to what interval and information it really captures.
Dude, it's free software so I shouldn't complain; I'll just have to play around with it, but I know there are better tools out there.