java.lang.OutOfMemoryError: unable to create new native thread

[Ganesh Narayan]

Hi

I am using a jboss v3.2.1 on a HP-UX 11.i V2 server.

We are getting the error
"java.lang.OutOfMemoryError: unable to create new native thread"

Could someone please advise why this error is occurring.

The server has 32 GB RAM allocated to it
We have allocated 2Gb heap size for the java process, 256 MB for the perm gen; thread stack size is 1mb

Could you also let us know the correct formula for the thread stack size, heap size that we should configure.

Below listed are existing kernel parameters:

max_thread_proc 3500 3500 Immed
ksi_alloc_max 33600 Default Immed
nproc 16388 16388 Immed
nkthread 17000 17000 Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
swchunk 4096 4096
maxuprc 256 Default Immed
msgmap 16390 (msgtql+2)msgmax 8192 Default Immed

msgtql 16388 nproc


Please let me know in case you need any other information.

Thanks,
Ganesh

[Dennis Handly]

What does "swapinfo -tam" show?
Also does kcusage(1m) show you nearing nkthread?
What are your values for maxdsiz and maxdsiz_64bit?

How many threads do you have?

[Steven E. Protter]

Shalom Ganesh,

Excessive memory use is one of the reasons that java continues to be updated on http://www.hp.com/go/java

There are recommended OS patches on the site, and I strongly recommend those.

You may have done nothing wrong. I'd like to see the response to Dennis' query.

SEP

[Ganesh Narayan]

Hi

Please find below the output of swapinfo

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 17408 0 17408 0% 0 - 1 /dev/vg00/lvol2
dev 16384 0 16384 0% 0 - 2 /dev/vg00/lvol10
reserve - 4197 -4197
memory 32700 6791 25909 21%
total 66492 10988 55504 17% - 0 -







================
Kcusage
================

hpapp040:/# kcusage
Tunable Usage / Setting
=============================================
dbc_max_pct 19 / 20
maxdsiz 142450688 / 2063835136
maxdsiz_64bit 46723072 / 7516192768
maxfiles_lim 236 / 10000
maxssiz 1179648 / 134217728
maxssiz_64bit 90112 / 1073741824
maxtsiz 757760 / 100663296
maxtsiz_64bit 2764800 / 2147483648
maxuprc 14 / 256
max_thread_proc 249 / 3500
maxvgs 4 / 10
msgmni 2 / 16388
msgseg 0 / 32767
msgtql 0 / 16388
nfile 2175 / 65536
nflocks 40 / 4096
ninode 2012 / 133152
nkthread 1123 / 17000
nproc 247 / 16388
npty 0 / 60
nstrpty 8 / 60
nstrtel 0 / 60
nswapdev 2 / 10
nswapfs 0 / 10
semmni 44 / 8192
semmns 102 / 16384
shmmax 32684560 / 12884901888
shmmni 48 / 1024
shmseg 24 / 1024



=========================
Maxdsize and maxdsize

maxdsiz 2063835136 2063835136 Immed
maxdsiz_64bit 0x1c0000000 0x1c0000000 Immed

==========================




---------------------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 8
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 800% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
java 24083 tndsrv 53.8 248 0.0 1.51gb 2.97gb STRMS


The value of thread count keeps fluctuating depending on the usage of the app

This snapshot was taken during quiescent period (low usage) of the app

Please let me know in case you need any other info

[Ganesh Narayan]

Thanks Steven for your reply.

Could you please let me know the specific patches you are referring to.

Thanks,
Ganesh

[Ganesh Narayan]

Hi Steven,

To add to the previous post... we are using Java 1.4
This system went live with the latest OS patches on Dec 2007.

There have been no updates provided by HP since Sep 06.

~ Ganesh

[Dennis Handly]

>This snapshot was taken during quiescent period (low usage) of the app

It would be better to take them near when it fails.

With a 2 Gb java heap, you don't have room for thread stacks. Can you use 64 bit java?

[Ganesh Narayan]

> Can you use 64 bit java?

Could you please explain this.

Is there any command which will show if i am running java version is 64bit/32bit

The HP-UX server is a 64bit
uname -a
HP-UX hpapp040 B.11.23 U ia64

[Dennis Handly]

You need to use -m64 to get 64 bit java. (That's assuming you have no JNIs.)

[Rajesh K Chaurasia]

Hi Ganesh:

What does java -version show for the JVM being used by your application? Are you using PA-RISC Java on Integrity under ARIES? Is so, the Java process may encounter the max. thread limit much earlier than expected due to limited heap size for ARIES. For more info and suggestion please refer

http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=c935c7b31d779110VgnVCM100000275d6e10RCRD#threads

Regards
-Rajesh

[Ganesh Narayan]

Hi Rajesh,

The command "java -version" shows
java -version
java version "1.4.2.13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2.13-070317-13:26)
Java HotSpot(TM) Server VM (build 1.4.2 1.4.2.13-070317-18:29-IA64N IA64, mixed mode)

Also we are not using Java under ARIES.

Ganesh

[Ganesh Narayan]

Hi Rajesh,

We are using an Itanium server.

Ganesh

[Ganesh Narayan]

Hi Denis,

Below is the output when the application was under load


hpapp040:/opt/TnDM/current/TnDTestServer/server/tndserver/log# kcusage
Tunable Usage / Setting
=============================================
dbc_max_pct 19 / 20
maxdsiz 159227904 / 2063835136
maxdsiz_64bit 46723072 / 7516192768
maxfiles_lim 282 / 10000
maxssiz 1179648 / 134217728
maxssiz_64bit 90112 / 1073741824
maxtsiz 757760 / 100663296
maxtsiz_64bit 2764800 / 2147483648
maxuprc 15 / 256
max_thread_proc 297 / 3500
maxvgs 4 / 10
msgmni 2 / 16388
msgseg 0 / 32767
msgtql 0 / 16388
nfile 2336 / 65536
nflocks 41 / 4096
ninode 2019 / 133152
nkthread 1241 / 17000
nproc 253 / 16388
npty 0 / 60
nstrpty 9 / 60
nstrtel 0 / 60
nswapdev 2 / 10
nswapfs 0 / 10
semmni 44 / 8192
semmns 102 / 16384
shmmax 32684560 / 12884901888
shmmni 48 / 1024
shmseg 24 / 1024

[Rajesh K Chaurasia]

Thanks Ganesh for the confirmation. This rules out ARIES max thread creation limit scenario.

You could try the application with 64-bit JVM (add -d64 to java command line). As Dennis has already suggested, this will not work if there are JNI libraries involved.

I would suggest finding the max thread usage in the application and computing the memory requirement based on stack and heap.

[Wim Rombauts]

Ganesh,

The message "java.lang.OutOfMemoryError: unable to create new native thread" means that the system 32 bit process has no more memory available to create a new OS-thread within this process.

May I ask why you have the current heap size, perm size and stack size ? Why I ask ? I once had a similar issue with a new java application that started crashing as soon as we were in production and the JVM got some load.
I started with increasing the maximum heap size and the stack size, and it only got worse.
Until I learned that the OS-process is more than the JVM heap size. By setting such huge heap size and stack size, the OS-porcess' memory usage rapidly grows beyond the 4GB maximum.
So I decreased the heap size, and went back to the default stack size. After that, I activated the "-Xmpas:on" option to allow maximum private memory for the java process.
Since then, the JVM is running fine and can easily take the load.

Also, with a program like glance or glanceplus (gpm), you can look at the memory consumption of the JVM as on OS-process, and also the number of threads you have.
With a program like HP-Jmeter, you can look inside the JVM to memory behaviour.
With these 2 tools you can better look at what the problem is : Is your JVM indeed in need of more memory, or has your JVM excess of memory, while there is nothing left for the OS-process housekeeping.