HPUX 11.00 has high User/Sys CPU usage with JRE 1.4.1

[Murugesan Guruswamy]

Hi,
Our application is a messaging software (Server Java application ) that includes lots of third party libraries ( JNI,C++ libraries). The application works fine in Windows and Solaris platforms for days. But on HPUX it goes into high cpu mode ( dual-cpu) within 5-6 hours of continuous load.

Our java version:
java version "1.4.1.00"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1.00-030116-09:58)
Java HotSpot(TM) Server VM (build 1.4.1 1.4.1.00-030116-11:27-PA_RISC2.0 PA2.0,
mixed mode)

We have a 2GB physical memory with 4-5GB of virtual memory. Our JVM Xmx size is also set to 1024m. We know that our JNI,C++ application is 32bit libraries and are limited to 1GB data space.

I am not sure if any of the latest os patches related to JDK 1.4.1 or HPUX 11 would cause this high cpu cycles on sched_yield() calls.
Anybody has insights?

I have uploaded information from my system that is stored in the zip file for the following:
tusc_1828* is first run
tusc_21708* is the second run.
vmstat* is the virtual memory.
sar* files are the system reports with the appropriate params.
swlist.txt contains the patches that I have applied on the system.

Thanks,
Murugesan

[James Murtagh]

Hi Murugesan,

Although I don't believe this to be the issue here, I think Java is optimised for a MxN thread model which I believe the later Solaris/Windows OS have. I think it was introduced in 11i V1.6 on hpux. Also, you didn't mention how many CPUs the others have.

What I do know is that for hpux you must keep up with the required (not recommended) patch set for Java. The main Java documentation for hpux is here:

http://www.hp.com/products1/unix/java/infolibrary/index.html

You will find a link to the required patches. To be fair you are at a very good patch level but there are a few you are missing. I also remember seeing issues with HotSpot performance but if I remember right it was mainly on 11i, I think related to thread pre-emption. However, I would advise contacting HP to confirm this.

From this page you will also see a lot of performance tools and white papers to optimise Java, if you haven't already used these then I would take a look.

Unfortunately I couldn't read the sar output you produced. The tusc analysis just looks like thread priority scheduling. Maybe if you could post the output of runable processes (ps -el|grep R) I could have a look at the relative priorities.

Regards,

James.

[Carlo Montanari]

You're spending all time in contention between threads.
What about garbage collection? You are running with verbose:gc, so the standard output (or error?) should contain statistics about GC's.
If you have glance, you can use it to retrieve measurements at the thread level.
Issuing a kill -3 to the jvm process, you can get a full thread dump of what is happening in the jvm (it doesn't kill the process, only request a dump); this is VERY useful in analisys.

[Steven E. Protter]

There are a lot of pre-requisite patches at http://www.hp.com/go/java

Are they all installed and configured correctly? I was amazed when I first discovered how many there were and how many dependencies they had.

Latest version on the tools is important.

What about the java plugin for netscape/mozilla? Is it installed. I found during the four hours I had it installed, it ramped the CPU up beyond belief.

I got rid of it FAST.

SEP