java.lang.OutOfMemoryError: Java heap space HP SIM 5.0 SP # 5

[SEBEMISMNUSA]

Far too often, these messages appear on the HP SIM web "console" http://hpsim:280, logged in fine, doing work, other times, HP SIM "just stops" monitoring (about 600 computer servers)

An error occurred connecting to the HP Systems Insight Manager
server:
java.lang.OutOfMemoryError: Java heap space
Click here to go to the login page.

Stop and restarting HP SIM get this system back online, monitoring and inventory computer servers . . but these errors keep occurring and disabling this HP SIM system.

Questions: Is there a fix for this (besides upgrading to HP SIM 5.1 sp # 1, BTW, this failed to work in January 2007 as listed at my posting at http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1091862

Software version and computer hardware information:

1. HP SIM version 5.0 sp # 5 or HP Systems Insight Manager Version C.05.00.02.00

2. Operating System: Microsoft Windows Server 2003, Standard Edition Service Pack 1

3. RDBMS - Microsoft SQL Server 2000 with sp # 4

4. Sun Java runtime or JRE that came with HP SIM 5.0 -
C:\Program Files\HP\Systems Insight Manager\j2re\bin>java -version
java version "1.5.0_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_01-b08)
Java HotSpot(TM) Client VM (build 1.5.0_01-b08, mixed mode)
C:\Program Files\HP\Systems Insight Manager\j2re\bin>

5. All Microsoft critical updates applies (I can provide the list if necessary)

6. Computer HP ProLiant DL360 G4; two Processor: Intel Xeon /3.40GHz with L2 1024 KB; 4 GB of Advanced ECC RAM, disks in raid with Smart Array 6i controller; firmware current, hardware running fine.

----------------------------------------------
Messages from various log files in the directory
C:\Program Files\HP\Systems Insight Manager\logs that were generated at the same time . . .
Exception in thread "MxLogSessionThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timeout guard" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MxRMIWatcher-secureRMI" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MxLogSessionThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MxLogSessionThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ScannerThread" java.lang.OutOfMemoryError: Java heap space
HP Systems Insight Manager shutting down: Lost connection to database, exception: java.sql.SQLException: Invalid state, the Connection object is closed.
JVM SHUTDOWNHOOK: Shutting down HP Systems Insight Manager...

----------------------------------------------
17:04:36,917 INFO [Server] Starting JBoss (MX MicroKernel)...
17:04:36,917 INFO [Server] Release ID: JBoss [WonderLand] 3.2.7 (build: CVSTag=JBoss_3_2_7 date=200501280217)
17:04:36,917 INFO [Server] Home Dir: C:\Program Files\HP\Systems Insight Manager\jboss
17:04:36,917 INFO [Server] Home URL: file:/C:/Program Files/HP/Systems Insight Manager/jboss/
17:04:36,917 INFO [Server] Library URL: file:/C:/Program Files/HP/Systems Insight Manager/jboss/lib/
17:04:36,917 INFO [Server] Patch URL: null
17:04:36,917 INFO [Server] Server Name: hpsim
17:04:36,917 INFO [Server] Server Home Dir: C:\Program Files\HP\Systems Insight Manager\jboss\server\hpsim
17:04:36,917 INFO [Server] Server Home URL: file:/C:/Program Files/HP/Systems Insight Manager/jboss/server/hpsim/
17:04:36,917 INFO [Server] Server Data Dir: C:\Program Files\HP\Systems Insight Manager\jboss\server\hpsim\data
17:04:36,917 INFO [Server] Server Temp Dir: C:\Program Files\HP\Systems Insight Manager\jboss\server\hpsim\tmp
17:04:36,917 INFO [Server] Server Config URL: file:/C:/Program Files/HP/Systems Insight Manager/jboss/server/hpsim/conf/
17:04:36,917 INFO [Server] Server Library URL: file:/C:/Program Files/HP/Systems Insight Manager/jboss/server/hpsim/lib/
17:04:36,917 INFO [Server] Root Deployment Filename: jboss-service.xml
17:04:36,917 INFO [Server] Starting General Purpose Architecture (GPA)...
17:04:37,167 INFO [ServerInfo] Java version: 1.5.0_01,Sun Microsystems Inc.
17:04:37,167 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.5.0_01-b08,Sun Microsystems Inc.
17:04:37,167 INFO [ServerInfo] OS-System: Windows 2003 5.2,x86
17:04:37,511 INFO [Server] Core system initialized
17:04:44,712 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
17:04:44,774 INFO [WebService] Using RMI server codebase: http://as071009:50013/
17:04:45,134 INFO [NamingService] Started jndi bootstrap jnpPort=50009, rmiPort=50008, backlog=50, bindAddress=/0.0.0.0, Client SocketFactory=null, Server SocketFactory=org.jboss.net.sockets.DefaultSocketFactory@ad093076
17:04:46,493 INFO [HPSIM_DEBUG] HP Systems Insight Manager Product Name: Systems Insight Manager 5.0 with SP5 - Windows
17:04:46,493 INFO [HPSIM_DEBUG] HP Systems Insight Manager Version C.05.00.02.00
17:04:46,493 INFO [HPSIM_DEBUG] HP Systems Insight Manager Build 2006-05-23 11:07
------------------------------------------------------------
Time when logged into http://hpinsightmonone.bemisco.net:280

@!@,2007-06-22 16:52:50 CDT,CONFIG,SUCCESS,ADD,QUERY,SUMMARY,,123334,1282403801,

@!@,2007-06-22 17:02:41 CDT,SESSION,SUCCESS,LOGIN,USER,Successful Login
(x.x.x.x),NOTE,,,,

---------------------------------------------

End

[SEBEMISMNUSA]

These are more error messages, when attempting to log into HP SIM;
HTTP Status 500 -
--------------------------------------------------------------------------------
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
javax.servlet.ServletException
org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:825)
org.apache.jasper.runtime.PageContextImpl.access$1100(PageContextImpl.java:64)
org.apache.jasper.runtime.PageContextImpl$12.run(PageContextImpl.java:745)
java.security.AccessController.doPrivileged(Native Method)
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:743)
org.apache.jsp.mxportal.home.intro_jsp._jspService(intro_jsp.java:274)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Unknown Source)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:272)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)
java.security.AccessController.doPrivileged(Native Method)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
sun.reflect.GeneratedMethodAccessor271.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Unknown Source)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:272)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:217)
root cause
java.lang.AssertionError
com.hp.mx.core.authorizationmanager.MxNodeToToolBoxSetTableForUser.add(MxNodeToToolBoxSetTableForUser.java:107)
com.hp.mx.core.authorizationmanager.MxAuthInfoQueryEngine.updateHashtables(MxAuthInfoQueryEngine.java:538)
com.hp.mx.core.authorizationmanager.MxAuthInfoQueryEngine.listNodeNames(MxAuthInfoQueryEngine.java:132)
com.hp.mx.core.authorizationmanager.MxAuthorizationManager.listNodeNamesForUser(MxAuthorizationManager.java:1015)
com.hp.mx.core.controllers.MxAuthController.getNodeNamesForUser(MxAuthController.java:2302)
com.hp.mx.core.controllers.MxToolController.getToolsForNonPrivilegedUser(MxToolController.java:3594)
com.hp.mx.core.controllers.MxToolController.getAuthorizedToolsForUser(MxToolController.java:2851)
com.hp.mx.core.controllers.MxToolController.getToolsForSessionUser(MxToolController.java:1643)
com.hp.mx.core.client.MxToolClientLocalImpl.getToolsForSessionUser(MxToolClientLocalImpl.java:372)
com.hp.mx.portal.settings.HomePageBean.isUserAuthorizedForTool(HomePageBean.java:543)
org.apache.jsp.mxportal.home.intro_jsp._jspService(intro_jsp.java:186)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Unknown Source)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:272)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:161)
java.security.AccessController.doPrivileged(Native Method)
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
sun.reflect.GeneratedMethodAccessor271.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:243)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAsPrivileged(Unknown Source)
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:272)
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:217)
note The full stack trace of the root cause is available in the Apache Tomcat/5.0.30 logs.
--------------------------------------------------------------------------------
Apache Tomcat/5.0.30

End

[David Claypool]

Any chance you ran out of hard disk space on that system? The only time I've seen things like this is after an application crash.

[SEBEMISMNUSA]

Regarding the question of disk space,
The C: partition has 2.4 GB of free space on a 10.5 GB partition. This contains the OS, MS SQL server software (but not the HP SIM database)
The E: partition has 18.6 GB free space on a 23.3 GB partition.
In addition, there are no messages in event viewer, from Microsoft Windows nor the HP Insight agents, that disk space is low. There are disk space thresholds set with the HP Insight agents on this computer.

The database are below their maximum size. The databases can automatically grow, too.

End

[SEBEMISMNUSA]

As a side note, the computer, the HP SIM application, and corresponding software (ex. MSSQL server) continued to work during these errors. There was a large discovery running (162,000 tcp/ip address scan), alert notifcations were sent.

Before rebooting this computer, to clear this error, mxconfigrepo -f found a few errors. The discovery had completed, too. The first mxconfigrepo pass cleared these errors. The second mxconfigrepo did not find any errors. I rebooted this computer. HP SIM is now running agian.

[SEBEMISMNUSA]

The HP SIM computer was rebooted at 1:05 AM. It and HP SIM restarted fine.

By 8:37 AM on the same day, it stopped. These messages appeared in a HP SIM error log DomainManager.####.out.log
in VCRepository.initialize - cannot find node mac like address - com.hp.mx.core.nodemanager.MxUndefinedNodeException: The name "mac like address" does not represent a node in this system.
in VCRepository.initialize - cannot find node mac like address - com.hp.mx.core.nodemanager.MxUndefinedNodeException: The name "mac like address" does not represent a node in this system.
***** SQLBase.initialize....

Starting Indication Listener on port: 50004
HP Systems Insight Manager started, be sure mxdtf is running or this will pause.
WBEM indications consumer 1
Exception in thread "ScannerThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1044]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IntervalTimer" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MxRMIWatcher-secureRMI" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1070]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "RMI RenewClean-[TCP/IP addr of HP SIM computer:1280]" java.lang.OutOfMemoryError: Java heap space

Results from mx.log just before this
@!@,2007-06-26 08:24:56 CDT,CONFIG,SUCCESS,ADD,NODE,SUMMARY, ,AS071009\mxadmin,0,12528,
@!@,2007-06-26 08:25:05 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12532,
@!@,2007-06-26 08:26:02 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12534,
@!@,2007-06-26 08:26:42 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12536,
@!@,2007-06-26 08:26:43 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12537,
@!@,2007-06-26 08:26:44 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12538,
@!@,2007-06-26 08:26:53 CDT,CONFIG,SUCCESS,ADD,NODE, ,SUMMARY,AS071009\mxadmin,0,12541,
@!@,2007-06-26 08:34:32 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12546,
@!@,2007-06-26 08:37:00 CDT,CONFIG,SUCCESS,MODIFY,NODE, ,SUMMARY,AS071009\mxadmin,0,12552
End

[LukaS_9]

Hi,
we have had a similar problem, but in a different environment (SIM 5.0.02 and VSE 2.5 on HPUX 11.11 CMS).
To fix it we have added the parameter "MX_JVM_OPTS = -XX:MaxPermSize=96m" into vseprefs.props CMS file. I exactly don't remember its default value in case of parameter absence (probably 64m).
After that we have stopped and restarted SIM.
You could set the parameter value as you want (128m...)

Probably this is not your case, but maybe you have to find and change the value of a specific java-parameter for SIM application.

regards
luca

[SEBEMISMNUSA]

Per LukaS suggestion, I made this change for a Microsoft Windows based HP SIM.
Added entry MX_JVM_OPTS = -XX:MaxPermSize=256m
in the file C:\Program Files\HP\Systems Insight Manager\config

Increased maximum memory allocation to 256 megabytes.

HP SIM appears to run faster at the :280 console. Time will tell how this works.

In addition, during the previous week,
with the mxnode -r command, I removed all printers, routers, switches, unknown and unmanaged computer and other devices from the HP SIM database. About five thousand entries (nodes) were removed. This, too, helped.

If there are any desirable Sun Java configuration settings, please let me know. The above setting appears to help.
End

[SEBEMISMNUSA]

This change, either 256 or 384M works very well. This, along with the Pegasus WMI 2.3 update, has fixed many issues I had with HP SIM 5.0 sp #5. HP SIM is finally working as I need after over year.

Added entry MX_JVM_OPTS = -XX:MaxPermSize=384m
in the file C:\Program Files\HP\Systems Insight Manager\config\Globalsettings.prop
Increased maximum memory allocation to 384 megabytes, most likley from 128 megabytes.

256m also worked.

End

[SEBEMISMNUSA]

This worked as previously mentioned.
Added entry MX_JVM_OPTS = -XX:MaxPermSize=384m
in the file C:\Program Files\HP\Systems Insight Manager\config\Globalsettings.prop
Increased maximum memory allocation to 384 megabytes, most likley from 128 megabytes.

End

[SEBEMISMNUSA]

Well, my last post did not work as well as I thought.

Further attempts
There is another file with Java settings
\Program Files\HP\Systems Insight Manager\javarun.bat
added these commands to the java -server command
-Xms768m -Xmx768m -Xmn384m

\Program Files\HP\Systems Insight Manager\config\globalsettings.props
Set
MX_JVM_OPTS=-XX:PermSize=512m -XX:MaxPermSize=512m

http://www.unixville.com/~moazam/stories/2004/05/17/maxpermsizeAndHowItRelatesToTheOverallHeap.html
http://forum.java.sun.com/thread.jspa?threadID=343789&forumID=27
http://java.sun.com/docs/hotspot/HotSpotFAQ.html

http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_oom

The VM prints "OutOfMemoryError" and exits. Increasing max heap size doesn't help. What's going on?

End of this posting

[SEBEMISMNUSA]

For the \Program Files\HP\Systems Insight Manager\config\globalsettings.prop file
this may be the answer
MX_JVM_OPTS=-XX:PermSize=128m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m

For the mxdomainmgr task to *completely* start, the Xmx and Xms need to be greater than 800m (for 800 megabytes). I set this to 1024m (1024 megagbytes or 1 Gigabyte)

If Xmx and Xms is set to0 small, ex. 256m, the mxdomainmgr process will start, but not completely; the mxinventory process will *not* start. HP SIM does not work.

Permsize & MaxPermsize, google on these terms for many Sun Java related articles.

End

[SEBEMISMNUSA]

For the \Program Files\HP\Systems Insight Manager\config\globalsettings.prop file
MX_JVM_OPTS=-XX:PermSize=128m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m

End