Java performance problems

[support_5]

Hi Folks,

Was wondering if anyone had any tips with regards to a performance problem I am having. On one of our 11i servers, we have a java application running (using HP's java v1.3.1). The symptom is that there is a high run queue (up to 6) on the machine, even though CPU usage at the time is low (around 5%). This is a dual CPU PA-RISC server with two 875 MHz cpu's, with 4GB ram. And I can't figure out what's causing the high run queue with a tiny CPU load. Also, this only seems to happen on one CPU, the other CPU seems to run normally.

When I look closer using gpm, I notice that when looking at the wait states of the java threads, the majority of the time spent waiting was on a resource labeled "system". Apparently this is a catch-all for other wait states that are either not common, or too small to warrant thier own wait state. Having a look at other servers around this site reveals that this wait state is hardly ever higher than about 15%, but on these processes, it is more like 85%!

Does anyone have any suggestions? Or has anyone seen this behaviour before?

(NB, I have run HPjconfig, and applied kernel mods and patches)

Thanks!

- Andrew Gray

[Massimo Bianchi · Accepted solution]

Hi,
unfortunatly I already saw this behavious on one of my customer, using a custom application (written in Basic).

I was able to alleviate the problem, applying some patches and some kernel tuning.


First: if you can , i suggest you to to to java 1.4.1, it's more stable and performant. See if it is advisable and if it help.

Kernel parameters: he had some unusually huge value here:

dbc_max_pct
(calculate to have 400Mb ram, no more)

ninode
fix it, let's say 4096
vx_ninode
fix it, let's say 3874 (usually 90%*nindoe)

these are the CACHE for hfs/vxfs

Other system paramters:
bufcache_hash_locks from 4096 to 2048
ftable_hash_locks from 4096 to 2048
vnode_cd_hash_locks from 4096 to 2048
vnode_hash_locks from 4096 to 2048


the reduction of this can alleviate the SYSTEM part.


PATCHES: here folows the list i used, for enabling fast vxfs file retriaval, avoiding some memory leak and so on:

PHKL_29110,
PHKL_25995, PHKL_25994, PHKL_25993, PHKL_28983 fast file description
PHKL_29047




PHKL_29611 - vx 3.5 mem leak
PHCO_29603 - vx 3.5 mem leak
PHCO_29495 - libc cumulative

###PHKL_27091, PHKL_27294, PHKL_27093 and PHKL_27094. sleep/wake up
PHKL_29707 (ex 27294)
PHKL_29706 (ex 27091)
PHKL_28238 (ex 27093)
PHKL_27094

PHKL_29115 performance degradation of jfs
PHKL_29527 performance buffercache
PHKL_29985 performance mmap
PHCO_29495 libc cumulative
PHKL_29611 - vx 3.5 mem leak
PHCO_29603 - vx 3.5 mem leak
PHNE_29211 - ONC/NFS general performance

HTH,
Massimo

[Massimo Bianchi]

I was forgetting: how do you start the java machine ?

Maybe it's a simple problem of parameters, maybe too many ram and the garbage collector of java it's not so smart....

Massimo

[support_5]

Hi,

Thanks for those tips. I'll go through them and post the results! Thanks!

- Andrew Gray

[Bill Hassell]

The runqueue is a measure of the number of processes running or waiting to run but no CPU is available. It is quite trivial to create some code which momentarily asks the kernel to do something (like provide the current date or open a file) and then terminate. Repeat this 500 times per second and the kernel will have a very high runqueue (I've seen 50 to 500) because of the large number of short-lived processes. The system state is not a catchall. Thye problem is that the state of a program can change dozens of times per second--humans are far too slow to comprehend what is happening. Glance is just trying to display a meaningful state within it's very long measurement window.


As far as the difference between different servers, unless they are all running the version of HP-UX, the same set of patches, the same kernel parameters and the same application code, it will be difficult to draw any conclusions.

[support_5]

Hi Bill, Thanks for the reply.

So what you're saying is that if it has a high run queue, it's not necessarily a bad thing? Especially if the cpu usage is still low?

But still, it would be good to know why it has a high run queue, don't you think? It's not like the example you gave because there aren't lots of short-lived processes running on the system, but it may be something else?

Any other ideas or suggestions?

Thanks!

- Andrew

[support_5]

Oh, for your information, attached is the java start up script with the command line options etc.

note I have added an option for performance monitoring temporarily (ie the -Xeprof option).

Thanks

- Andrew Gray

[Bill Hassell]

> So what you're saying is that if it has a high run queue, it's not necessarily a bad thing? Especially if the cpu usage is still low?

That's correct. It does indicate that the processes spend the majority of their time waiting on system calls. An example would be a program that calls select() in a tight loop (bad programming technique). There will be high system overhead and a bunch of these programs (or threads) will create a high runqueue but not consume a lot of CPU time.

> But still, it would be good to know why it has a high run queue, don't you think? It's not like the example you gave because there aren't lots of short-lived processes running on the system, but it may be something else?

Yes, figuring out what is happening will be useful. Java threads are particularly tricky to debug since some of the code can be in libraries. On the other hand, the system is probably fairly responsive, it may not be worth the effort to track down the reason(s).

[support_5]

Thanks Bill, appreciate that input. I'll still apply those kernel parameters and patches suggested by Massimo Bianchi. I am implementing them tonight.

One other question I have is why it would only ever be on the first CPU on a dual CPU system?? Seems strange to me? Is it a symptom of bad programming, or should we have more JVM's running? What do people think?

Thanks again.

- Andrew Gray

[Bill Hassell]

The scheduling algorithm is fairly straightforward but because the processes are generating system calls and very little runtime, the monarch processor (the primary CPU as defined by the kernel) will tend to get most of the processing. Once long term processes start running (something that computes for several secs/mins) then CPU 2 will get busy. Try this simple shell code to create 100% CPU usage:

while :
do
:
done

You can start 2 or 3 of these at the same time to see how the system allocates CPU usage. Since the Java code lasts for such a short time, you should not see much of a difference in response time.