topic Re: OutOfMemoryError : unable to create new native thread in Operating System - Linux

OutOfMemoryError : unable to create new native thread

Varun_5 — Wed, 19 Apr 2006 08:09:19 GMT

Hi,
I am using SunOne Application Server, and i am executing some test cases testing failover of session ( with server resatrt ) with entity Bean, HomeHandled reference stored in the session.
While executing my testcases started failing.
When i saw into the server log i found that it failing with following error complaining java.lang.OutOfMemoryError: unable to create new native thread.

Following is my system configuration :
Operating System : HP-UX B11.11
Architecture : PA_RISC
No. Of Processor : 4
Total Physical Memory : 4,192,256 kbyte
Free Physical Memory : 1,752,436 kbyte
Total Swap Space : 4,194,304 kbyte
Free Swap Space : 4,194,304 kbyte
Java Version : 1.5.0_01
I have Out Of Box installed on my machine.
Some pf the kernel parameter of Machine is :
max_thread_proc = 3000
maxdsiz = 2063835136
maxdsiz_64bit = 1073741824
maxfiles = 2048
maxfiles_lim = 2048
maxssiz = 8388608
maxssiz_64bit = 8388608
maxswapchunk = 2048
maxtsiz 67108864
maxtsiz_64bit = 1073741824
maxurpc = 75
maxuser = 512
maxvgs = 10
nproc = 2048
and ulimit is unlimited.

Error seen in server.log
==========================
[#|2006-04-19T16:49:50.994+0530|WARNING|sun-appserver-ee8.1|javax.enterprise.sys
tem.stream.err|_ThreadID=352;|java.lang.OutOfMemoryError: unable to create new n
ative thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.j
ava:833)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SS
LSocketImpl.java:1025)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.
java:619)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.ja
va:59)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65
)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198
)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getDefaultDomain(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.c
heckConnection(RMIConnector.java:1543)
at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(Clien
tCommunicatorAdmin.java:165)
at java.lang.Thread.run(Thread.java:595)
========================================
Please help me to understand the issue.
I cant see any problem with system configuration also.

Thanks,
Varun

Re: OutOfMemoryError : unable to create new native thread

Peter Godron — Wed, 19 Apr 2006 08:20:21 GMT

Varun,
is your maxdsiz_64bit correct?
Less than maxdsiz ?

Re: OutOfMemoryError : unable to create new native thread

Varun_5 — Wed, 19 Apr 2006 08:58:11 GMT

yes peter,

Value of maxdsiz_64bit on my machine is 1073741824
maxdsiz = 2063835136
maxdsiz_64bit = 1073741824

Re: OutOfMemoryError : unable to create new native thread

Peter Godron — Wed, 19 Apr 2006 09:06:03 GMT

Varun,
sorry that I did not make myself clear.
I think your maxdsiz_64bit is actually too low and should at least the same as maxdsiz.

I also found:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=211942 which seems to point in the same direction.

Re: OutOfMemoryError : unable to create new native thread

Varun_5 — Wed, 19 Apr 2006 09:37:05 GMT

Peter,
I have gone through that mail thread but it doesn't look like solving the stated problem. Is there any chance that test application or application server can be suspected ?

Re: OutOfMemoryError : unable to create new native thread

Sameer_Nirmal — Thu, 20 Apr 2006 01:13:29 GMT

Hi Varun,

Does the test case failure creates "core" file or "hs_err*.log" files?

Whats the patch level of the system?
Are the required patches for the Java version applied?

If the "core" file is being generated, you can use WDB or GDB to examine it.

You try reducing the thread stack size using "-Xss" option or increasing the heap size.

You might want to use "HPjconfig" tool to verify the kernel tunnables.

Refer the following link for details
http://www.hp.com/products1/unix/java/?jumpid=go/java

Re: OutOfMemoryError : unable to create new native thread

Varun_5 — Thu, 20 Apr 2006 07:49:30 GMT

Hi Sameer,
I will use HPjconfig and try to figure out if some more kernel parameter can be tunned for jvm.
I will keep posting about the progress.

Thanks,
Varun

Re: OutOfMemoryError : unable to create new native thread

Seth Kaplan — Tue, 23 Jan 2007 13:27:57 GMT

Did you ever get this resolved? We're getting a similar error using JBOSS.

Seth

Re: OutOfMemoryError : unable to create new native thread

Bill Hassell — Tue, 23 Jan 2007 17:54:08 GMT

Just to clarify:

> maxdsiz = 2063835136
> maxdsiz_64bit = 1073741824

is quite wrong! If maxdsiz_64 is less than maxdsiz, then both parameters are silently enforced to the maxdsiz_64 value. ALWAYS set maxdsiz_64 larger than maxdsiz.

As far as OutOfMemory, there are many, many causes but generally it has nothing to do with physical memory. HP-UX is a virtual memory system so available memory is a combination of RAM plus swap space. But there are several limits that are much less than virtual memory. Kernel parameter maxdsiz sets the maximum local data (heap) for a 32 bit process. Parameter maxdsiz_64 refers only to 64 bit processes. shmmax refers to the largest shared memory segment allowed. If a set of programs need a larger segment, this kernel parameter must be adjusted.

To see all the ulimit values, use ulimit -a. If this fails, your shell is badly out of date (ksh needs a patch). Note also that shared memory (32 bit programs) must be mapped in a single space and can be heavily fragmented with InterProcess Comunication (IPC) objects like semaphores and other shared memory segments.

You may have a massively oversized kernel (normal is 8 to 15 megs) due to an oversized parameter, or your buffer cache is far too large (should be 600to 1200 megs).