topic Re: java.lang.OutOfMemoryError: unable to create new native thread in Operating System - HP-UX

java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Tue, 29 Jul 2003 19:45:19 GMT

Hi,

When I try to start weblogic 702 on HP-UX B.11.00 U 9000/800, I get following exception:
Starting WebLogic Server...
<140005> tion /usr4/apps/bea/user_projects/GlobalSight/./config.xml>
<090082> ng using realm myrealm.>
<000327> ogic Admin Server "GlobalSightServer" for domain "GlobalSight">
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at weblogic.kernel.ExecuteThreadManager.startThreads(ExecuteThreadManage
r.java:177)
at weblogic.kernel.ExecuteThreadManager.setThreadCount(ExecuteThreadMana
ger.java:158)
at weblogic.kernel.ExecuteThreadManager.(ExecuteThreadManager.java
:81)
at weblogic.kernel.Kernel.addExecuteQueue(Kernel.java:368)
at weblogic.kernel.Kernel.addExecuteQueue(Kernel.java:358)
at weblogic.jms.dispatcher.DispatcherImpl.(DispatcherImpl.java:63)
at weblogic.jms.dispatcher.DispatcherManager.(DispatcherManager.ja
va:57)
at weblogic.jms.dispatcher.DispatcherManager.create(DispatcherManager.ja
va:114)
at weblogic.jms.JMSService.initializeCommon(JMSService.java:377)
at weblogic.jms.JMSServiceServerLifeCycleImpl.initialize(JMSServiceServe
rLifeCycleImpl.java:54)
at weblogic.t3.srvr.ServerLifeCycleList.initialize(ServerLifeCycleList.j
ava:54)
at weblogic.t3.srvr.T3Srvr.initialize1(T3Srvr.java:782)
at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:594)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:282)
at weblogic.Server.main(Server.java:32)
<040014> , JMS is shutting down...>
<000361> ed during initialization of server subsystem ==> weblogic.jms.JMSServiceServerLi
feCycleImpl@1b7dfa>

===============================================
At present, I have following parameter values set on the HP-UX machine:
Parameter Value
===============================================nproc (20+8*MAXUSERS)
maxusers 400
max_thread_proc 2000
nkthread (((NPROC*7)/4)+16)
maxfiles_lim 2048
maxfiles 1024
nfile (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
ncallout (16+NPROC)

Any help is appreciated.

Thanks & Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

A. Clay Stephenson — Tue, 29 Jul 2003 19:55:23 GMT

I would increase maxdsiz, possibly maxssiz, and perhaps (though doubtful) maxtsiz. If this is 64-bit code then increase the 64bit (e.g. maxdsiz_64bit) counterparts and build a new kernel. You could also conceivably be running low on virtual memory so that an increase in swap space might be needed.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Tue, 29 Jul 2003 20:01:04 GMT

Thanks Stephensen,

I have following values for the mentioned parameters:
# kmtune -q maxdsiz
Parameter Value
===============================================maxdsiz 0x04000000
maxssiz 0x00800000
maxtsiz 0x04000000

How can I increase these values?

Regards,

VIlas

Re: java.lang.OutOfMemoryError: unable to create new native thread

A. Clay Stephenson — Tue, 29 Jul 2003 20:09:27 GMT

Your current 32bit sizes for maxdsiz, maxssiz, and maxtsiz are 64MB, 8MB, and 64MB respectively. I would increase maxdsiz to 256MB, maxssiz to 32MB, and maxtsiz to 128MB --- these are just suggestions and if you have lots of memory thenm feel free to increase maxdsiz to 512MB or so but I wouldn't increase maxssiz beyond 64MB. Again, your application may be 64-bit code so you may need to increase the 64-bit values -- if your OS is a 64-bit version. The easiest way to do this is make the changes in SAM and let it build and install a new kernel. SAM will automatically reboot when ready.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Tue, 29 Jul 2003 21:45:21 GMT

Hi Stephenson,

I increased the values of the parameters which you have mentioned, but I'm getting the same error. I set these values using 'kmtune' command. I tried using sam, but while creating new kernal, it gives some error. Please check the attached screenshot for the error.

Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

sdip — Wed, 30 Jul 2003 08:49:15 GMT

Vilas,

Please change the following parameters.

dbc_max_pct 15
dbc_min_pct 5
max_thread_proc 1024
maxdsiz 2 GB
maxdsiz_64bit 3GB
maxfiles 4096
maxfiles_lim 4096
maxssiz 250 MB
maxtsiz_64bit 350MB
maxuprc 2048
msgmnb 65535
msgmax 32767
msgmni 2048
msgseg 32767
msgssz 1024
msgtql 2048
nfile 30000
nlocks 15000
ninode 15000

The above parameters will help you to rectify the problem.

As per you last reply that config error message due to some config file which I cant recall now. You may need to take help of local HP engg for it.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Wed, 30 Jul 2003 21:37:55 GMT

Hi,

I set all the mentioned parameters, except 'nlocks', but no luck. There is no parameter by name 'nlocks'.

Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

James Murtagh — Wed, 30 Jul 2003 22:24:43 GMT

Hi,

You really need to use a tool named "tusc" for a case like this. We need to see the system calls and error codes to determine the problem. Looking at the log it appears the error happens during thread creation. Threads within a process should share the same text regions....I think the stack is mapped with each thread so it appears to be a data (or heap) sizing issue, hence maxdsiz (if it is indeed a 32 bit process, see previous responses). Also, please be wary or changing 20 - odd kernel parameters in one go even if directed by HP personel.

Tusc for PA can be found here:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,2894,00.html

Please use the following options to begin with when starting the application (and post the output if possible) :

tusc -p -v -f -l -u

Also check out the Java page for hpux which lists the required!! patches and certain performance and analytical tools.

Please also provide the output of:

# /usr/sbin/swapinfo -tam
# ipcs -mob

for completeness.

Regards,

James.

Re: java.lang.OutOfMemoryError: unable to create new native thread

James Murtagh — Wed, 30 Jul 2003 22:28:09 GMT

Hi again,

Sorry, I forgot, that parameter is nflocks and when you say "no luck" do you mean no luck after the kernel has been rebuilt or when building the new kernel? If the latter is the case do the following:

# cd /stand/build
# cp ../system .
# mk_kernel -s system

Please post the results.

Regards,

James.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Thu, 31 Jul 2003 15:34:17 GMT

Re: java.lang.OutOfMemoryError: unable to create new native thread

James Murtagh — Thu, 31 Jul 2003 18:11:47 GMT

Hi Vilas,

OK, there is no point looking at the tusc or VM stuff as you have more serious issues. The unsatisfied symbols suggest you have re-installed PHKL_18543. This is called a LITS (line in the sand) patch and should not be re-installed normally, as it will revert your system back to that patch level. You have a few options here:

1) Use "swlist -l product|grep PHKL" and note all kernel patches numerically after 18543. You can then download these and their dependencies from the ITRC and re-install. You can do this in one bundle to save any problems with multiple kernel rebuilds.
2) Use /usr/contrib/bin/check_patches to confirm the analysis and the kernel patches required. If you don't have it you can download and install patch PHCO_27779, no reboot required.

If you are unsure please ask, or post the output from the check_patches utility.

Also, kmtune at 11.00 is just an inetrface to editing the system file. You will not have modified your kernel in any way doing this while not running mk_kernel and kmupdate. Please use SAM if in doubt.

Once you have installed the kernel patches you will need to test the application again.

Regards,

James.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Bob Bean — Thu, 31 Jul 2003 18:54:50 GMT

I recently had a similar problem, which went unresolved for weeks.
I can't tell you how many kernel changes I made, but I couldn't get by it.
I finally ended up installing WL8.1, and it resolved the issue.
Not sure if this will work for you, but it might.

Good luck-
Stace

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Mon, 04 Aug 2003 16:18:37 GMT

Hi James,

First of all thanks for your valuable help. There are around 80-85 patches after 'PHKL_18543'. So, do I need to download all these 80-85 patches or is there any easy way to get them downloaded in a bundle or to get this problem fixed? Also, the utility 'check_patches' is not there; So, I will have to download this utility and apply patch 'PHCO_27779', right?

Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

Paddy_1 — Mon, 04 Aug 2003 16:57:13 GMT

Vilas,

Typically OutOfMemory means that the JVM has run out the heap size allocated for it.

There are parameters like
-Xms set initial Java heap size. example: -Xms32m
-Xmx set maximum Java heap size. example: -Xmx64m
-Xmn set new generation Java heap size. example: -Xmn64m
-Xss set Java thread stack size. example: -Xss128k

You might want to pass appropriate value by saying

java -Xms64 -Xms128 ... in the startup script.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Mon, 04 Aug 2003 17:07:57 GMT

James,

I forgot to attach swlist output. Pleas check the attachment for the required patches.

Paddy, I tried setting heap sizes in the weblogic start script, but it didn't work.

Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Mon, 04 Aug 2003 20:10:46 GMT

Hi James,

Here is the output of the check_patches. Please check the attachment.

Thanks & Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

James Murtagh — Mon, 04 Aug 2003 21:58:04 GMT

Hi Vilas,

Sorry, just back online after an odd night shift. It appears you have indeed re-installed PHKL_18543 - to any other forum member not in the know do NOT do this before contacting HP or seeking guidance from the more experienced forum guys here. You will have to re-install ALL kernel patches numerically after 18543 to get your system back in a normal state I'm afraid. You can download all the patches in a single bundle from the ITRC. All you need to do is go here:

http://www4.itrc.hp.com/service/patch/search.do?pageContextName=hpux:::

For each kernel patch in turn add this to the selected patch list. Just hit back on your browser and keep adding the patches, it will store your previous selections. Note this will be a big download so make sure you are referencing your nearest site.

Regards,

James.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Tue, 05 Aug 2003 17:45:23 GMT

Hi James,

Our System Admin started downloading the patches but he says, "When I find the patches from ITRC, and the recommended patch is different from the one I'm finding, I'm supposed to download the recommended one." And further he says, "I downloaded some of the patches (from PHKL-18798 to PHKL-20016) as 'patches1.tar'. When I tried to install them, it stops it because of the error in the kernel. How am I to install these patches if we keep running into this problem regarding the kernel?"

Thanks & Regards,

Vilas

Re: java.lang.OutOfMemoryError: unable to create new native thread

James Murtagh — Tue, 05 Aug 2003 18:36:38 GMT

Hi Vilas,

As you are trying to recover the system I would just download the exact patches you already have. You can make a note of any you feel need to be superseded at a later stage. It is important you download all the patches in one bundle to save any kernel rebuild issues. I would exclude PHKL_18543 from the download list as it is large and we know the problem starts here, it does not need to be re-installed. For the swinstall set the reinstall option to true. Once all have been installed your kernel should be rebuilt correctly and you can the reboot.

regards,

James.

Re: java.lang.OutOfMemoryError: unable to create new native thread

Vilas — Thu, 07 Aug 2003 14:54:59 GMT

Hi James,

Our System Admin finished with applying patches. Attached is the output of 'check_patches' utility. So, is it ok to proceed with this? Also, now 'mk_kernel' command doesn't give any error.

Thanks & Regards,

Vilas