java.lang.OutOfMemoryError: unable to create new native thread

[Ganesh Narayan]

Hi Rajesh,

The command "java -version" shows
java -version
java version "1.4.2.13"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2.13-070317-13:26)
Java HotSpot(TM) Server VM (build 1.4.2 1.4.2.13-070317-18:29-IA64N IA64, mixed mode)

Also we are not using Java under ARIES.

Ganesh

[Ganesh Narayan]

Hi Rajesh,

We are using an Itanium server.

Ganesh

[Ganesh Narayan]

Hi Denis,

Below is the output when the application was under load


hpapp040:/opt/TnDM/current/TnDTestServer/server/tndserver/log# kcusage
Tunable Usage / Setting
=============================================
dbc_max_pct 19 / 20
maxdsiz 159227904 / 2063835136
maxdsiz_64bit 46723072 / 7516192768
maxfiles_lim 282 / 10000
maxssiz 1179648 / 134217728
maxssiz_64bit 90112 / 1073741824
maxtsiz 757760 / 100663296
maxtsiz_64bit 2764800 / 2147483648
maxuprc 15 / 256
max_thread_proc 297 / 3500
maxvgs 4 / 10
msgmni 2 / 16388
msgseg 0 / 32767
msgtql 0 / 16388
nfile 2336 / 65536
nflocks 41 / 4096
ninode 2019 / 133152
nkthread 1241 / 17000
nproc 253 / 16388
npty 0 / 60
nstrpty 9 / 60
nstrtel 0 / 60
nswapdev 2 / 10
nswapfs 0 / 10
semmni 44 / 8192
semmns 102 / 16384
shmmax 32684560 / 12884901888
shmmni 48 / 1024
shmseg 24 / 1024

[Rajesh K Chaurasia]

Thanks Ganesh for the confirmation. This rules out ARIES max thread creation limit scenario.

You could try the application with 64-bit JVM (add -d64 to java command line). As Dennis has already suggested, this will not work if there are JNI libraries involved.

I would suggest finding the max thread usage in the application and computing the memory requirement based on stack and heap.

[Wim Rombauts]

Ganesh,

The message "java.lang.OutOfMemoryError: unable to create new native thread" means that the system 32 bit process has no more memory available to create a new OS-thread within this process.

May I ask why you have the current heap size, perm size and stack size ? Why I ask ? I once had a similar issue with a new java application that started crashing as soon as we were in production and the JVM got some load.
I started with increasing the maximum heap size and the stack size, and it only got worse.
Until I learned that the OS-process is more than the JVM heap size. By setting such huge heap size and stack size, the OS-porcess' memory usage rapidly grows beyond the 4GB maximum.
So I decreased the heap size, and went back to the default stack size. After that, I activated the "-Xmpas:on" option to allow maximum private memory for the java process.
Since then, the JVM is running fine and can easily take the load.

Also, with a program like glance or glanceplus (gpm), you can look at the memory consumption of the JVM as on OS-process, and also the number of threads you have.
With a program like HP-Jmeter, you can look inside the JVM to memory behaviour.
With these 2 tools you can better look at what the problem is : Is your JVM indeed in need of more memory, or has your JVM excess of memory, while there is nothing left for the OS-process housekeeping.