Java 1.4 on HP-UX 11.11 signal 11 exception

[Andres Reiner]

While running a java application we received the following exception and stack trace:

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0xC0000000014163FC
Function=CPXPxdelrc
Library=/opt/cms/lib/libcplexadapter90_64.sl

Current Java thread:
"VBJ ThreadPool Worker" daemon prio=2 tid=800000010025fca8 nid=126 lwp_id=121548 runnable [0x800003ff3f801000..0x800003ff3f800880]
at com.prosrm.optimizer.engine.solver.cplex.jni.CplexLink.lpopt(Native Method)
at com.prosrm.optimizer.engine.solver.cplex.CplexSolver.solve(CplexSolver.java:307)
at com.prosrm.optimizer.engine.processor.DefaultRequestProcessor.process(DefaultRequestProcessor.java:164)
at com.prosrm.optimizer.engine.RequestManager.handleOptimize(RequestManager.java:150)
at com.prosrm.optimizer.engine.RequestManager.handleRequest(RequestManager.java:96)
at com.prosrm.optimizer.engine.impl.OptimizerEngine.optimize(OptimizerEngine.java:99)
at com.prosrm.pro.common.AbstractBaseOptimizerRule.optimize(AbstractBaseOptimizerRule.java:95)
at com.prosrm.pro.common.AbstractBaseOptimizerRule.apply(AbstractBaseOptimizerRule.java:65)
at com.prosrm.rules.impl.RuleSet.executePlan(RuleSet.java:462)
at com.prosrm.rules.impl.RuleSet.apply(RuleSet.java:420)
at com.prosrm.rules.impl.RuleProcessor.applyWithRuleSetAndPlan(RuleProcessor.java:321)
at com.prosrm.rules.impl.RuleProcessor.applyWithRuleSetAndPlan(RuleProcessor.java:355)
at com.prosrm.rules.corba.RuleProcessorImpl.applyWithRuleSetAndPlanAndProperties(RuleProcessorImpl.java:389)
at com.prosrm.corbagen.ruleprocessserver.RuleProcessorPOATie.applyWithRuleSetAndPlanAndProperties(RuleProcessorPOATie.java:187)
at com.prosrm.corbagen.ruleprocessserver.RuleProcessorPOA._invoke(RuleProcessorPOA.java:233)
at com.prosrm.corbagen.ruleprocessserver.RuleProcessorPOA._invoke(RuleProcessorPOA.java:58)
at com.inprise.vbroker.poa.POAImpl.invoke(POAImpl.java:2693)
at com.inprise.vbroker.poa.ActivationRecord.invoke(ActivationRecord.java:109)
at com.inprise.vbroker.GIOP.GiopProtocolAdapter.doRequest(GiopProtocolAdapter.java:824)
at com.inprise.vbroker.IIOP.ServerProtocolAdapter.doRequest(ServerProtocolAdapter.java:68)
at com.inprise.vbroker.GIOP.GiopProtocolAdapter.dispatchMessage(GiopProtocolAdapter.java:1106)
at com.inprise.vbroker.orb.TPDispatcherImpl$TPDispatcher.run(TPDispatcherImpl.java:100)
at com.inprise.vbroker.orb.ThreadPool$PoolWorker.run(ThreadPool.java:76)

Dynamic libraries:
/opt/java1.4/bin/PA_RISC2.0W/java
text:0x4000000000000000-0x400000000000a454 data:0x8000000100000000-0x8000000100000be8

/usr/lib/pa20_64/libpthread.1
text:0xc0000000001a0000-0xc0000000001b8118 data:0x800003ffe7800000-0x800003ffe7804530

/opt/java1.4/jre/lib/PA_RISC2.0W/server/libjvm.sl
text:0xc000000000400000-0xc0000000010faa58 data:0x800003ffe7400000-0x800003ffe7729518

/usr/lib/pa20_64/libdl.1
text:0xc00000000003c000-0xc00000000003d964 data:0x800003ffe6c00000-0x800003ffe6c001a8

/usr/lib/pa20_64/libc.2
text:0xc000000000200000-0xc00000000031be18 data:0x800003ffe6800000-0x800003ffe681b320

/usr/lib/pa20_64/libm.2
text:0xc0000000000d0000-0xc0000000000f5d48 data:0x800003ffe6400000-0x800003ffe6402bd0

/usr/lib/pa20_64/librt.2
text:0xc0000000000a8000-0xc0000000000ac194 data:0x800003ffe6000000-0x800003ffe6000448

/usr/lib/pa20_64/libcl.2
text:0xc000000000100000-0xc000000000184ea8 data:0x800003ffe5800000-0x800003ffe584b4e8

/usr/lib/pa20_64/libCsup.2
text:0xc0000000000b0000-0xc0000000000c6cc0 data:0x800003ffe5400000-0x800003ffe5403d70

/opt/java1.4/jre/lib/PA_RISC2.0W/native_threads/libhpi.sl
text:0xc0000000001c0000-0xc0000000001cf1a0 data:0x800003ffe4800000-0x800003ffe4801d90

/opt/java1.4/jre/lib/PA_RISC2.0W/libverify.sl
text:0xc0000000001d0000-0xc0000000001e39c8 data:0x800003ffe3c00000-0x800003ffe3c01810

/opt/java1.4/jre/lib/PA_RISC2.0W/libjava.sl
text:0xc000000000320000-0xc00000000034d6e8 data:0x800003ffe3400000-0x800003ffe3405e30

/opt/java1.4/jre/lib/PA_RISC2.0W/libzip.sl
text:0xc000000000350000-0xc0000000003663cc data:0x800003ffe3000000-0x800003ffe3002180

/opt/java1.4/jre/lib/PA_RISC2.0W/libnet.sl
text:0xc000000000370000-0xc00000000038205c data:0x800003ff42c00000-0x800003ff42c00cf8

/lib/pa20_64/libnm.sl
text:0xc0000000001e4000-0xc0000000001ef4b4 data:0x800003ff42400000-0x800003ff424006b0

/opt/java1.4/jre/lib/PA_RISC2.0W/libioser12.sl
text:0xc000000000198000-0xc00000000019d124 data:0x800003ff41800000-0x800003ff418003c0

/usr/lib/pa20_64/libnss_files.1
text:0xc000000000188000-0xc0000000001906ac data:0x800003ff41000000-0x800003ff41000de8

/opt/cms/lib/libcplexadapter90_64.sl
text:0xc000000001100000-0xc00000000141d278 data:0x800003ff3e000000-0x800003ff3e0165b0

/lib/pa20_64/libnsl.1
text:0xc000000001440000-0xc0000000014c04e4 data:0x800003ff3dc00000-0x800003ff3dc111d8

/usr/lib/pa20_64/libxti.2
text:0xc000000000390000-0xc0000000003a6e6c data:0x800003ff3d800000-0x800003ff3d802e30


Heap at VM Abort:
Heap
def new generation total 737280K, used 557547K [800003ff46800000, 800003ff78800000, 800003ff78800000)
eden space 655360K, 85% used [800003ff46800000, 800003ff6887ac38, 800003ff6e800000)
from space 81920K, 0% used [800003ff6e800000, 800003ff6e800000, 800003ff73800000)
to space 81920K, 0% used [800003ff73800000, 800003ff73800000, 800003ff78800000)
tenured generation total 1638400K, used 1620672K [800003ff78800000, 800003ffdc800000, 800003ffdc800000)
the space 1638400K, 98% used [800003ff78800000, 800003ffdb6b0208, 800003ffdb6b0400, 800003ffdc800000)
compacting perm gen total 24320K, used 24125K [800003ffdc800000, 800003ffddfc0000, 800003ffe0800000)
the space 24320K, 99% used [800003ffdc800000, 800003ffddf8f690, 800003ffddf8f800, 800003ffddfc0000)

Local Time = Thu Jun 30 11:16:44 2005
Elapsed Time = 82430
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.4.2 1.4.2.03-040401-22:03-PA_RISC2.0W PA2.0W (aCC_AP) mixed mode)
#
# An error report file has been saved as hs_err_pid7850.log.
# Please refer to the file for further information.
#

The hs_err_pid7850.log file contained the same information as listed above.
The core file was deleted before it could be retrieved.

Would upgrading to a newer JDK have any affect on this issue?

[mark rozhin]

> Would upgrading to a newer JDK have any affect on this issue?

Maybe not. You are blowing up in the native method lpopt called from CplexSolver.java:307. I would read through that code first.

[Daavid Turnbull]

You have not had much of a response so I will share my limited insight.

Sig 11 is a seg fault, usually caused when the process attempts to access data which does not belong to it. Typical causes include faulty pointers, pointers running off the end of an array, corrupted stack (which can be caused for example by running over a local variable on the stack).

I would look closely at what data you are passing in particularly in the form of arrays and structures.

Another area to look at is data you are passing to your threads. If an alternate thread releases data that another thread uses that could also cause a seg fault.

Are they underlying native code functions thread safe?

If possible, you may wish to over allocate data arrays before you pass them on for diagnostic purposes. If the problem changes or goes away it may help you narrow down the cause.

good luck __daavid

[Andres Reiner]

The problem was not related to our JVM version but instead was related to a data condition in the code.