Server Management - Systems Insight Manager
1748266 Members
3410 Online
108760 Solutions
New Discussion юеВ

Re: After upgrade, continued Java problems.. help!!

 
Brian Hahne
Regular Advisor

After upgrade, continued Java problems.. help!!

Upgraded to 5.3 and was having issues with the system locking up. It would let me input credentials, but never get to a page. Java out of memory errors in the system, and a post here, led me to add this:
MX_JVM_OPTS=-XX:PermSize=128m -XX:MaxPermSize=128m -Xms1024m -Xmx1024m

to the globalsettings.props file.

Came in to work this morning, noticed I had no SIM alerts since last night.
Looked in log.
I see the following now (large paste):
4 Feb 22:28:28,412 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 99% used
04 Feb 22:28:32,912 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 99% used
04 Feb 22:28:35,849 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 100% used
04 Feb 22:28:40,381 ERROR [AbstractServiceEndpointServlet] Error processing web service request
ClientAbortException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at org.jboss.ws.core.server.ServiceEndpointManager.processRequest(ServiceEndpointManager.java:485)
at org.jboss.ws.core.server.AbstractServiceEndpointServlet.doPost(AbstractServiceEndpointServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at org.jboss.ws.core.server.AbstractServiceEndpointServlet.service(AbstractServiceEndpointServlet.java:75)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at sun.reflect.GeneratedMethodAccessor754.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:283)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at sun.reflect.GeneratedMethodAccessor755.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1173)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1185)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:43)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:764)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
... 50 more
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:166)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1518)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1485)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1449)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1394)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:64)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:764)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.PrintWriter.flush(PrintWriter.java:270)
at java.io.PrintWriter.flush(PrintWriter.java:270)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:489)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:366)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:366)
at org.jboss.ws.core.utils.DOMWriter.print(DOMWriter.java:215)
at org.jboss.ws.core.soap.SOAPFaultImpl.writeElement(SOAPFaultImpl.java:732)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElementContent(SOAPElementImpl.java:834)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElement(SOAPElementImpl.java:819)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElementContent(SOAPElementImpl.java:834)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElement(SOAPElementImpl.java:819)
at org.jboss.ws.core.soap.SOAPElementWriter.writeElementInternal(SOAPElementWriter.java:149)
at org.jboss.ws.core.soap.SOAPElementWriter.writeElement(SOAPElementWriter.java:130)
at org.jboss.ws.core.soap.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:291)
at org.jboss.ws.core.server.ServiceEndpointManager.sendResponse(ServiceEndpointManager.java:517)
at org.jboss.ws.core.server.ServiceEndpointManager.processRequest(ServiceEndpointManager.java:481)
... 45 more
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:283)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:272)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:666)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
... 73 more
04 Feb 22:28:43,818 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 99% used
04 Feb 22:28:55,740 ERROR [[ResourceAccessAPIsBean]] Servlet.service() for servlet ResourceAccessAPIsBean threw exception
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:283)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:272)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:666)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:764)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at java.io.PrintWriter.flush(PrintWriter.java:270)
at java.io.PrintWriter.flush(PrintWriter.java:270)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:489)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:366)
at org.jboss.ws.core.utils.DOMWriter.printInternal(DOMWriter.java:366)
at org.jboss.ws.core.utils.DOMWriter.print(DOMWriter.java:215)
at org.jboss.ws.core.soap.SOAPFaultImpl.writeElement(SOAPFaultImpl.java:732)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElementContent(SOAPElementImpl.java:834)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElement(SOAPElementImpl.java:819)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElementContent(SOAPElementImpl.java:834)
at org.jboss.ws.core.soap.SOAPElementImpl.writeElement(SOAPElementImpl.java:819)
at org.jboss.ws.core.soap.SOAPElementWriter.writeElementInternal(SOAPElementWriter.java:149)
at org.jboss.ws.core.soap.SOAPElementWriter.writeElement(SOAPElementWriter.java:130)
at org.jboss.ws.core.soap.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:291)
at org.jboss.ws.core.server.ServiceEndpointManager.sendResponse(ServiceEndpointManager.java:517)
at org.jboss.ws.core.server.ServiceEndpointManager.processRequest(ServiceEndpointManager.java:481)
at org.jboss.ws.core.server.AbstractServiceEndpointServlet.doPost(AbstractServiceEndpointServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at org.jboss.ws.core.server.AbstractServiceEndpointServlet.service(AbstractServiceEndpointServlet.java:75)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at sun.reflect.GeneratedMethodAccessor754.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:283)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at sun.reflect.GeneratedMethodAccessor755.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:218)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:393)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
04 Feb 22:28:55,740 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 100% used
04 Feb 22:28:58,912 INFO [HPSIM_DEBUG] [Panic Logger-0] (Construction Cache for HardwareStatusPolling:42) HW Polling null object was null
04 Feb 22:28:58,912 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 100% used
04 Feb 22:29:03,396 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 100% used

The errors continue on...
Leading to more:

04 Feb 23:15:14,258 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 100% used
04 Feb 23:15:17,211 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 99% used
04 Feb 23:15:21,680 INFO [HPSIM_DEBUG] [Panic Logger-0] (Low Memory Detector) Heap collection hitting upper limits 99% used
04 Feb 23:15:25,086 ERROR [AbstractServiceEndpointServlet] Error processing web service request

Then lots of these later in the night:
05 Feb 04:05:23,797 WARN [SpyMessageConsumer] Message listener com.hp.isee.event.EventNotificationListener@1e300ed threw a throwable.
java.lang.OutOfMemoryError: Java heap space
05 Feb 04:05:23,797 WARN [SpyMessageConsumer] Message listener com.hp.isee.event.EventNotificationListener@1e300ed threw a throwable.
java.lang.OutOfMemoryError: Java heap space
05 Feb 04:05:23,797 WARN [SpyMessageConsumer] Message listener com.hp.isee.event.EventNotificationListener@1e300ed threw a throwable.
java.lang.OutOfMemoryError: Java heap space

and then some:
05 Feb 04:40:01,948 ERROR [JIoEndpoint] Socket accept failed
java.lang.OutOfMemoryError: Java heap space
05 Feb 04:40:07,699 ERROR [JIoEndpoint] Socket accept failed
java.lang.OutOfMemoryError: Java heap space
05 Feb 04:40:30,996 ERROR [JIoEndpoint] Socket accept failed
java.lang.OutOfMemoryError: Java heap space

More errors, then rebooted at 9am, and it's back up.

HELP!!!!!!!!
12 REPLIES 12
JKytsi
Honored Contributor

Re: After upgrade, continued Java problems.. help!!

What is your SIM HW ?
cpus/memory/etc..
Remember to give Kudos to answers! (click the KUDOS star)

You can find me from Twitter @JKytsi
Brian Hahne
Regular Advisor

Re: After upgrade, continued Java problems.. help!!

BL20P G4, mirrored 146Gb drives
4Gb Ram, dual 3.20 Ghz Processors.

Windows 2003 OS, SQL 2005.
JKytsi
Honored Contributor

Re: After upgrade, continued Java problems.. help!!

Doh ..Now I get those too.

Windows 2003 enterprise ed. 16GB RAM / 2 x opterons DL385G1 ~3000 managed items
Remember to give Kudos to answers! (click the KUDOS star)

You can find me from Twitter @JKytsi
SJO EGGER
Regular Advisor

Re: After upgrade, continued Java problems.. help!!

morning

also the same problem ......
HP told me to set the parameter "JVMMAXHEAP=1200" in the globalsettings.props but this does not help. call is still open - i will write here if i get some new information from HP and please do the same.

thank you
roland
Brian Hahne
Regular Advisor

Re: After upgrade, continued Java problems.. help!!

Please do. This is a bug they need to fix.
Right now all we can do is reset the SIM server every 12-18 hours.
Ragarc
Occasional Advisor

Re: After upgrade, continued Java problems.. help!!

Same here:

BL460c G1, 16Gb Ram, 2 quad 2.6 Ghz Processors with Windows 2003 and SQL 2005.
Scott White_5
Advisor

Re: After upgrade, continued Java problems.. help!!

Same here - every 2 days or so...

DL380 G5
4GB RAM
Windows 2003 x32 Std.
Remote SQL 2005 DB

I find it hard to believe HP didn't run across this in testing before release - does HP dogfood SIM internally before releasing new versions?

The java heap errors + the ILO2 failure on identification problem have me wishing I had not rolled 5.3 out.

HP, please help us...

Thanks,
Scott
Rob Buxton
Honored Contributor

Re: After upgrade, continued Java problems.. help!!

Our 5.3 environment lasted an entire week! we can't be working it as hard as you folk.

Hung this morning though.

I'm watching this thread with interest.
Rob Buxton
Honored Contributor

Re: After upgrade, continued Java problems.. help!!

Also to add, we start getting Java errors that indicate an issue with VMM.
Caused by: com.hp.mx.vmm.api.v3_6.APIException: Virtual machine not found.

After quite a while later we get the low memory / out of memory errors.