topic Re: Java - JVM large heap size problem in Operating System - HP-UX

Java - JVM large heap size problem

Morgan Pyne — Tue, 17 Jun 2003 07:01:21 GMT

Re: Java - JVM large heap size problem

Alex Glennie — Tue, 17 Jun 2003 07:58:06 GMT

point to note : I'm not a java programmer.

OK that over with I've found the following which may help ....

Making the heap larger decreases the frequency of collections but makes each collection take longer. One good rule of thumb is that you should ensure that there is more than 50% free space available in the heap after a collection. With the -verbosegc flag on, it will print the amount of free space after each collection.

For server applications where swap space and physical memory are not concerns, it may be better to run with a very large heap to spread out the collections. For applications that have a stable heap resident set (i.e., they have a consistent amount of live data after each collection), you can set the initial and maximum heap sizes to be the same value. Setting the heap sizes to be 3-4 times the resident set will put the amount of free heap space in the 65-75% range.

Increasing the startup heap size will cause more system swap to be used. The user will have to balance heap requirements against swap usage. The -ms setting should be high enough that the system is not frantically doing garbage collection, but low enough that system resources are not being consumed too eagerly.

For example, your application needs between 12 and 16 MB. You would want to specify:

-ms32m -mx32m

This would prevent the JVM from trying to fit your app into 1 MB or 4 MB, and would also provide plenty of free space to improve gc performance.

For applications that have a widely varying resident set (or have a different resident set on different runs), try to leave the initial heap size at the default and set the maximum value large enough to handle the largest data set. An example of this kind of application is the Java compiler itself. When compiling small files, it doesn't use much heap space. When compiling lots of files (or very large ones) it uses a lot of space. By keeping the initial heap size low, it doesn't overuse the physical memory of the machine when compiling small files. The tradeoff is that compiling large files will cause the collector to go through some spasms in order to grow the heap to its desired size.

The default is 1 MB.

Re: Java - JVM large heap size problem

Morgan Pyne — Tue, 17 Jun 2003 12:25:20 GMT

Hi Alex,

Thx, for the reply. Useful info regarding heap management, however I can't give you any points since you didn't actually address my problem/question at all :-(

Cheers,
Morgan

Re: Java - JVM large heap size problem

Jeff Schussele — Tue, 17 Jun 2003 12:30:22 GMT

Hi Morgan,

How much swap space do you have configured?
Swap is not only used to page out processes, it's also needed to reserve space "in case" it needs to page out.
When you start that java process, it will need to reserve that much swap space.

Post output of:

swapinfo -tam

Rgds,
Jeff

Re: Java - JVM large heap size problem

Jeff Schussele — Tue, 17 Jun 2003 12:36:31 GMT

Oh & I forgot to mention that your maxdsiz_64bit MUST be larger than 2 Gb as well.
You have to be able to fit this data size into BOTH swap & the maxdsiz_64bit space.

Rgds,
Jeff

Re: Java - JVM large heap size problem

Massimo Bianchi — Tue, 17 Jun 2003 13:02:03 GMT

Hi,
doing some seach i always find that thery are using

-XmsSIZEm -XmxSIZEm

using equal SIZE values.

Instead i see that you specify different values (for min and max).

Did you try specifyng the same value, just to see if it is an application error when these value differ ? Maybe with 1500m each.

Just a try...

HHTH,
Massimo

Re: Java - JVM large heap size problem

Elena Leontieva — Tue, 17 Jun 2003 13:46:52 GMT

Have you seen this document yet?
I am not sure if it pertains to yiour case.

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000066903282

Re: Java - JVM large heap size problem

Morgan Pyne — Tue, 17 Jun 2003 13:58:22 GMT

Hi Jeff, Massimo

Here is the output of swapinfo -tam :

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 3000 0 3000 0% 0 - 2 /dev/vg00/lvol14
dev 3000 0 3000 0% 0 - 2 /dev/vg00/lvol15
reserve - 337 -337
memory 2628 68 2560 3%
total 9652 405 9247 4% - 0 -

and FYI, here are the current settings for the tunable kernel parameters (from /stand/system) :

STRMSGSZ 65535
create_fastlinks 1
dbc_max_pct 15
dbc_min_pct 2
default_disk_ir 1
fs_async 1
max_thread_proc 3000
maxdsiz 2063835136
maxdsiz_64bit 0X400000000
maxfiles 2048
maxfiles_lim 2048
maxssiz 0X04FB3000
maxssiz_64bit 0X10000000
maxswapchunks 4096
maxtsiz 0X40000000
maxtsiz_64bit 0X100000000
maxuprc 256
maxusers 512
ncallout 6000
nfile 30000
ninode 4000
nkthread 6000
nproc 2068
npty 200
nstrpty 200
shmmax 0X40000000
vps_ceiling 64

AFAIK, the java-out-of-the-box utility set these to high enough sensible values.
(maxdsiz_64bit is at 16gig)

Do you notice any problems with these settings ?

Do you attach any significance to the fact that the error is reporting an attempt to allocate a negative number of bytes?

Massimo:

Changing the Xmx and Xms values to be the same does not affect the behaviour in any way.

Regards & thanks,
Morgan

Re: Java - JVM large heap size problem

Jeff Schussele — Tue, 17 Jun 2003 14:41:37 GMT

Re: Java - JVM large heap size problem

Morgan Pyne — Tue, 17 Jun 2003 15:06:31 GMT

Hi Jeff,

Thanks - I have read these installation notes also, but I was under the impression that this change was only necessary if you are embedding libjvm in a native application (e.g. if linking libjvm to a user-developed program, it would be necessary to chatr the resulting binary). Or am I misreading the release notes ?

I have already verified the patch level requirements and appear to be compliant.

Cheers,
Morgan

Re: Java - JVM large heap size problem

Morgan Pyne — Tue, 17 Jun 2003 15:50:56 GMT

Me again :-)

Just re-reading this thread and thought I'd better add some further info to clarify things.

Although in my original postings I referred to attempting to use Xmx2000m, I should note that this error is also reproduceable for me with Xmx as low as 1400m (which is even below the 1500-2000MB range discussed in the release notes above).

Can somebody confirm what the maximum allowed heap size should be for invoking the 64bit jvm in my environment?

Morgan

Re: Java - JVM large heap size problem

Mike Stroyan — Wed, 18 Jun 2003 17:33:13 GMT

It looks like 64 bit java 1.4 has some interesting problems. I confirmed that -Xmx values from -Xmx1605m to -Xmx4040m cause trouble.
Even bigger values like -Xmx5000m are OK. That can even exceed available swap, since the jvm mmaps that really big area with a MAP_NORESERVE flag to delay allocating swap.
I went back to trying 'smaller' values and they sometimes worked as well.
There are no clearly responsible failed system calls in tusc traces of the failing cases.
It seems that the jvm is using some bad rule for deciding memory allocation success, perhaps related fitting regions into open address range areas determined by examining pstat_procvm results.

By the way, the maxdsiz values do not limit the size of -Xmx settings. The maxdsiz limits are for the malloc/sbrk heap. The -Xmx limit is for the java's mmap based heap. Mmap regions don't count towards maxdsiz.

Re: Java - JVM large heap size problem

Mike Stroyan — Wed, 18 Jun 2003 18:24:11 GMT

Looking at a few more test cases, it seems that your use of -Xincgc is more closely tied to the problem than any particular -Xmx value. Could you just use the default generational garbage collector?

Re: Java - JVM large heap size problem

Morgan Pyne — Thu, 19 Jun 2003 11:23:22 GMT

Hi Mike,

This is indeed interesting news. Unfortunately, I am not in a position to test this right now as I am at a conference and the machine in question is hosting part of the conference website.

I will try your suggestions once the machine is free and post my results here (it will probably be next week). Meanwhile, if you find out any further information on the topic I would love to hear the details. Is this a known bug?

Thx and regards,
Morgan

Re: Java - JVM large heap size problem

Morgan Pyne — Thu, 19 Jun 2003 14:14:33 GMT

Mike,

The conference just finished and I managed to sneak in some quick tests before the machines get shut down and shipped.

You were absolutely spot on the ball - removing the incremental garbage collection allowed me to specify any value for the Xmx , tested all the way up to 3000m.

Thx a lot and well spotted. Should have kept my points awards till after I tested so sorry about that. If you want to post another 1 line reply I'll award you 10 points for it!

I guess there should be a bug report created for this?

Cheers and all the best,
Morgan

Re: Java - JVM large heap size problem

Mike Stroyan — Thu, 19 Jun 2003 16:34:09 GMT

I already filed a defect against this. A lab engineer is having a look at it.

Re: Java - JVM large heap size problem

Mike Stroyan — Wed, 25 Jun 2003 20:48:15 GMT

The lab engineer reports that this seems to be a manifestation of Javasoft bug id 4867781.