java.lang.OutOfMemoryError: unable to create new native thread

[Vilas]

Hi,

When I try to start weblogic 702 on HP-UX B.11.00 U 9000/800, I get following exception:
Starting WebLogic Server...
<140005> tion /usr4/apps/bea/user_projects/GlobalSight/./config.xml>
<090082> ng using realm myrealm.>
<000327> ogic Admin Server "GlobalSightServer" for domain "GlobalSight">
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at weblogic.kernel.ExecuteThreadManager.startThreads(ExecuteThreadManage
r.java:177)
at weblogic.kernel.ExecuteThreadManager.setThreadCount(ExecuteThreadMana
ger.java:158)
at weblogic.kernel.ExecuteThreadManager.(ExecuteThreadManager.java
:81)
at weblogic.kernel.Kernel.addExecuteQueue(Kernel.java:368)
at weblogic.kernel.Kernel.addExecuteQueue(Kernel.java:358)
at weblogic.jms.dispatcher.DispatcherImpl.(DispatcherImpl.java:63)
at weblogic.jms.dispatcher.DispatcherManager.(DispatcherManager.ja
va:57)
at weblogic.jms.dispatcher.DispatcherManager.create(DispatcherManager.ja
va:114)
at weblogic.jms.JMSService.initializeCommon(JMSService.java:377)
at weblogic.jms.JMSServiceServerLifeCycleImpl.initialize(JMSServiceServe
rLifeCycleImpl.java:54)
at weblogic.t3.srvr.ServerLifeCycleList.initialize(ServerLifeCycleList.j
ava:54)
at weblogic.t3.srvr.T3Srvr.initialize1(T3Srvr.java:782)
at weblogic.t3.srvr.T3Srvr.initialize(T3Srvr.java:594)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:282)
at weblogic.Server.main(Server.java:32)
<040014> , JMS is shutting down...>
<000361> ed during initialization of server subsystem ==> weblogic.jms.JMSServiceServerLi
feCycleImpl@1b7dfa>

===============================================
At present, I have following parameter values set on the HP-UX machine:
Parameter Value
===============================================nproc (20+8*MAXUSERS)
maxusers 400
max_thread_proc 2000
nkthread (((NPROC*7)/4)+16)
maxfiles_lim 2048
maxfiles 1024
nfile (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NSTRPTY+NSTRTEL))
ncallout (16+NPROC)


Any help is appreciated.

Thanks & Regards,

Vilas

[A. Clay Stephenson]

I would increase maxdsiz, possibly maxssiz, and perhaps (though doubtful) maxtsiz. If this is 64-bit code then increase the 64bit (e.g. maxdsiz_64bit) counterparts and build a new kernel. You could also conceivably be running low on virtual memory so that an increase in swap space might be needed.

[Vilas]

Thanks Stephensen,

I have following values for the mentioned parameters:
# kmtune -q maxdsiz
Parameter Value
===============================================maxdsiz 0x04000000
maxssiz 0x00800000
maxtsiz 0x04000000

How can I increase these values?

Regards,

VIlas

[A. Clay Stephenson]

Your current 32bit sizes for maxdsiz, maxssiz, and maxtsiz are 64MB, 8MB, and 64MB respectively. I would increase maxdsiz to 256MB, maxssiz to 32MB, and maxtsiz to 128MB --- these are just suggestions and if you have lots of memory thenm feel free to increase maxdsiz to 512MB or so but I wouldn't increase maxssiz beyond 64MB. Again, your application may be 64-bit code so you may need to increase the 64-bit values -- if your OS is a 64-bit version. The easiest way to do this is make the changes in SAM and let it build and install a new kernel. SAM will automatically reboot when ready.

[Vilas]

Hi Stephenson,

I increased the values of the parameters which you have mentioned, but I'm getting the same error. I set these values using 'kmtune' command. I tried using sam, but while creating new kernal, it gives some error. Please check the attached screenshot for the error.

Regards,

Vilas

[sdip]

Vilas,

Please change the following parameters.

dbc_max_pct 15
dbc_min_pct 5
max_thread_proc 1024
maxdsiz 2 GB
maxdsiz_64bit 3GB
maxfiles 4096
maxfiles_lim 4096
maxssiz 250 MB
maxtsiz_64bit 350MB
maxuprc 2048
msgmnb 65535
msgmax 32767
msgmni 2048
msgseg 32767
msgssz 1024
msgtql 2048
nfile 30000
nlocks 15000
ninode 15000

The above parameters will help you to rectify the problem.

As per you last reply that config error message due to some config file which I cant recall now. You may need to take help of local HP engg for it.

[Vilas]

Hi,

I set all the mentioned parameters, except 'nlocks', but no luck. There is no parameter by name 'nlocks'.

Regards,

Vilas

[James Murtagh]

Hi,

You really need to use a tool named "tusc" for a case like this. We need to see the system calls and error codes to determine the problem. Looking at the log it appears the error happens during thread creation. Threads within a process should share the same text regions....I think the stack is mapped with each thread so it appears to be a data (or heap) sizing issue, hence maxdsiz (if it is indeed a 32 bit process, see previous responses). Also, please be wary or changing 20 - odd kernel parameters in one go even if directed by HP personel.

Tusc for PA can be found here:

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,2894,00.html

Please use the following options to begin with when starting the application (and post the output if possible) :

tusc -p -v -f -l -u

Also check out the Java page for hpux which lists the required!! patches and certain performance and analytical tools.

Please also provide the output of:

# /usr/sbin/swapinfo -tam
# ipcs -mob

for completeness.

Regards,

James.

[James Murtagh]

Hi again,

Sorry, I forgot, that parameter is nflocks and when you say "no luck" do you mean no luck after the kernel has been rebuilt or when building the new kernel? If the latter is the case do the following:

# cd /stand/build
# cp ../system .
# mk_kernel -s system

Please post the results.

Regards,

James.

[James Murtagh · Accepted solution]

Hi Vilas,

OK, there is no point looking at the tusc or VM stuff as you have more serious issues. The unsatisfied symbols suggest you have re-installed PHKL_18543. This is called a LITS (line in the sand) patch and should not be re-installed normally, as it will revert your system back to that patch level. You have a few options here:

1) Use "swlist -l product|grep PHKL" and note all kernel patches numerically after 18543. You can then download these and their dependencies from the ITRC and re-install. You can do this in one bundle to save any problems with multiple kernel rebuilds.
2) Use /usr/contrib/bin/check_patches to confirm the analysis and the kernel patches required. If you don't have it you can download and install patch PHCO_27779, no reboot required.

If you are unsure please ask, or post the output from the check_patches utility.

Also, kmtune at 11.00 is just an inetrface to editing the system file. You will not have modified your kernel in any way doing this while not running mk_kernel and kmupdate. Please use SAM if in doubt.

Once you have installed the kernel patches you will need to test the application again.

Regards,

James.

[Bob Bean]

I recently had a similar problem, which went unresolved for weeks.
I can't tell you how many kernel changes I made, but I couldn't get by it.
I finally ended up installing WL8.1, and it resolved the issue.
Not sure if this will work for you, but it might.

Good luck-
Stace

[Vilas]

Hi James,

First of all thanks for your valuable help. There are around 80-85 patches after 'PHKL_18543'. So, do I need to download all these 80-85 patches or is there any easy way to get them downloaded in a bundle or to get this problem fixed? Also, the utility 'check_patches' is not there; So, I will have to download this utility and apply patch 'PHCO_27779', right?


Regards,

Vilas

[Paddy_1]

Vilas,

Typically OutOfMemory means that the JVM has run out the heap size allocated for it.

There are parameters like
-Xms set initial Java heap size. example: -Xms32m
-Xmx set maximum Java heap size. example: -Xmx64m
-Xmn set new generation Java heap size. example: -Xmn64m
-Xss set Java thread stack size. example: -Xss128k


You might want to pass appropriate value by saying

java -Xms64 -Xms128 ... in the startup script.

[Vilas]

James,

I forgot to attach swlist output. Pleas check the attachment for the required patches.

Paddy, I tried setting heap sizes in the weblogic start script, but it didn't work.

Regards,

Vilas

[Vilas]

Hi James,

Here is the output of the check_patches. Please check the attachment.

Thanks & Regards,

Vilas

[James Murtagh]

Hi Vilas,

Sorry, just back online after an odd night shift. It appears you have indeed re-installed PHKL_18543 - to any other forum member not in the know do NOT do this before contacting HP or seeking guidance from the more experienced forum guys here. You will have to re-install ALL kernel patches numerically after 18543 to get your system back in a normal state I'm afraid. You can download all the patches in a single bundle from the ITRC. All you need to do is go here:

http://www4.itrc.hp.com/service/patch/search.do?pageContextName=hpux:::

For each kernel patch in turn add this to the selected patch list. Just hit back on your browser and keep adding the patches, it will store your previous selections. Note this will be a big download so make sure you are referencing your nearest site.

Regards,

James.

[Vilas]

Hi James,

Our System Admin started downloading the patches but he says, "When I find the patches from ITRC, and the recommended patch is different from the one I'm finding, I'm supposed to download the recommended one." And further he says, "I downloaded some of the patches (from PHKL-18798 to PHKL-20016) as 'patches1.tar'. When I tried to install them, it stops it because of the error in the kernel. How am I to install these patches if we keep running into this problem regarding the kernel?"

Thanks & Regards,

Vilas

[James Murtagh]

Hi Vilas,

As you are trying to recover the system I would just download the exact patches you already have. You can make a note of any you feel need to be superseded at a later stage. It is important you download all the patches in one bundle to save any kernel rebuild issues. I would exclude PHKL_18543 from the download list as it is large and we know the problem starts here, it does not need to be re-installed. For the swinstall set the reinstall option to true. Once all have been installed your kernel should be rebuilt correctly and you can the reboot.

regards,

James.

[Vilas]

Hi James,

Our System Admin finished with applying patches. Attached is the output of 'check_patches' utility. So, is it ok to proceed with this? Also, now 'mk_kernel' command doesn't give any error.

Thanks & Regards,

Vilas

[James Murtagh]

Hi Vilas,

The output of check_patches indicates you have fully recovered from the PHKL_18543 re-install. Is Java still giving the error?

Regards,

James.

[Vilas]

Hi James,

After applying patches, I tried starting weblogic and it still gives the same error. But when I checked some kernel parameters, they are set at the default values. So, which parameters should I set or should I follow the parameter values suggested by others? Also, check_patches output shows two errors; so do we need to apply those two patches?

Thanks & Regards,

Vilas

[James Murtagh]

Hi Vilas,

My apologies, yes you will need to re-install PHKL_23193 and PHKL_24011. Just ensure PHKL_18543 is not in the depot with them, we don't want a repeat of what happened.

Also, while you are doing this ensure you have the required Java patches for the version you are using:

http://www.hp.com/products1/unix/java/patches/index.html

If you click on the bundles it mentions the required patches will be listed. You can find your version of java using this command:

# swlist -l product|grep -i java

I think in terms of the kernel parameters we will take Clay's recommendations on the 32 bit maxtsiz, maxdsiz and maxssiz. However, as you haven't posted the 64 bit equivalents I am reluctant to advise changing those. If you want me to have a look post the output from this command:

# kmtune|grep max[tsd]siz_64bit

To change the values and rebuild the kernel using the following steps:

# cd /stand/build
# /usr/lbin/sysadm/system_prep -s system
# kmtune -s maxdsiz=268435456 -S system
# kmtune -s maxssiz=33554432 -S system
# kmtune -s maxdsiz=134217728 -S system
# mk_kernel -s system

(if complete with no errors)

# cp ../system ../system.prev
# cp ../vmunix ../vmunix.prev
# kmupdate

You can then reboot the system:

# cd /
# shutdown -ry 0

Cheers,

James.

[James Murtagh]

D'oh! Obviously the third kmtune should be changing the text limit....

# kmtune -s maxtsiz=134217728 -S system

spot who cuts and pastes.

[Vilas]

Hi James,

Our System Admin has problem downloading/installing PHKL_23193 and PHKL_24011 patches. He says,
"when I select to download PHKL-23193/PHKL_24011, PHKL-18543 is part of the dependencies". But we are not suppose to reinstall patch 18543. So he further says, "but how can you install 23193 if 18543 is required; Is it possible just to not select it when I download the patches, because I can't "remove" it from the patch list."

Thanks & Regards,

Vilas