Swapiness or not to swapiness - that is the question

[Michael Steele_2]

I have a new java application that takes 20 hours to perform the same number of transactions as its predecessor did. The old box did it in one hour. The new box does it in 20 hours.

The application people are screaming about process heap size and I don't see VSZ or RSS growing over time for any process.

The current box has a 2.6 kernel; I don't know what the previous box was.

I've seen one solution to reduce swapiness down from the default of 60 down to 20 to force cache to be flushed more frequently.

Used swap in 'free -m' is zero.

Comments?

[Alzhy]

What version of Linux sir?

If you have access to the old environment, is there someting in /etc/security/limits.conf that is specific to the user account that the Java app runs as?

[Matti_Kurkela · Accepted solution]

Which version of Java?

Have you already checked the JVM's memory parameters?

With modern versions of the Sun-developed JVM, they are set with the -Xms and -Xmx command line options when starting the Java application, and the default maximum for Java heap size is 1/4 of the system memory or 1 GB, whichever is _smaller_.

You say swap usage is zero and you don't see VSZ or RSS growing over time: that makes me suspect the application's memory usage might hit the JVM's limit pretty soon after it's started and then stay at that level "forever".

Since the JVM enforces the limit, from the OS's viewpoint it looks like the combination of application+JVM is not requesting any more memory, although the application within the JVM could certainly use more.

You don't mention anything that would indicate even the slightest memory pressure at the OS level: in fact, the zero swap usage would indicate that there has been *no* memory pressure since the last time the system was rebooted. I strongly suspect this is a JVM-level problem.

Does the Java application output any messages to stdout/stderr? If the JVM is hitting the ulimits, it usually tells that loud and clear.

MK

[TwoProc]

There is actually LOTS you can do to tune a Java engine. Unfortunately, unless you've got a nice background in it, it seems to be similar to a dark art. :-)

As Matti said, check your startup params and make sure you've got enough space in that engine for it to run.

I *can* recommend someone to you, that I know has had great sucess tuning Java environments - however, it's not permitted here - so you'd have to expose an email address or other that I could send mail to. If you don't want to expose your own, consider one of the email providers who do temp mailboxes.

[DeafFrog]

Hi ,

there is a thread and dump analysis tool available on IBM site , which can analyse the core generated by kill -3 Java-PID in gui mode.
it will seprately identify the java threads and reason for their wait state if any .

Reg ,

[dirk dierickx]

swapiness has nothing to do with flushing the cache. it is a parameter on how much the kernel should cache.

for example, if you set it to 10, the kernel will try not to swap a lot and keep as much as possible in memory, if you set it to 90 almost everything will be swapped out except the running process.

changing the swappiness will most likely not help here unless this java process starts a lot of other things along the way.

i can recommend visual-vm, take a look at it or give it to your developers for troubleshooting;

https://visualvm.dev.java.net/features.html