JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

[Carlo Henrico_1]

Hi All

I have been running JBOSS on the above for a while now and the customer decided to upgrade to Oracle 10. System was reconfigured, SHLIB_PATH and LD_LIBRARY_PATH reset as well as ORACLE_HOME. However now when I start JBOSS I get the following:

Java HotSpot(TM) Server VM warning: Unexpected Signal 11 occured under user-defined signal handler 0x75abf0ea
Stack_Trace: error while unwinding stack
( 0) 0xcf41d804 report_and_die__7VMErrorFv + 0x154 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 1) 0xcf2f7850 JVM_handle_hpux_signal__Q2_2os4HpuxSFiP9__siginfoPvT1 + 0xaf8 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 2) 0xcf2f332c signalHandler__Q2_2os4HpuxSFiP9__siginfoPv + 0x4c [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 3) 0xccc1a398 xehInterpretSavedSigaction + 0x2d8 [/opt/mqm/lib/libmqmcs_r.sl]
( 4) 0xccc1a678 xehExceptionHandler + 0x1c8 [/opt/mqm/lib/libmqmcs_r.sl]
( 5) 0xc0213280 _sigreturn [/usr/lib/libc.2]
( 6) 0xd36f57f8 BN_set_word + 0x10 [/u01/app/oracle/10g/lib32/libnnz10.sl]
( 7) 0xd3732a80 BN_mod_inverse + 0xd0 [/u01/app/oracle/10g/lib32/libnnz10.sl]
( 8) 0xd38fe3b8 BN_MONT_CTX_set_locked + 0x1d8 [/perago/lib/libpersecj.sl]
( 9) 0xd38b0464 RSA_eay_public_decrypt + 0x254 [/perago/lib/libpersecj.sl]
(10) 0xd3878fc4 RSA_public_decrypt + 0x24 [/perago/lib/libpersecj.sl]
(11) 0xd38b0f40 RSA_verify + 0x190 [/perago/lib/libpersecj.sl]
(12) 0xd390a7f0 EVP_VerifyFinal + 0x128 [/perago/lib/libpersecj.sl]
(13) 0xd38c255c ASN1_item_verify + 0x114 [/perago/lib/libpersecj.sl]
(14) 0xd38864ec X509_CRL_verify + 0x2c [/perago/lib/libpersecj.sl]
(15) 0xd3b34fd8 loadX509CRLBase64__11CertHandlerFP6MemBufP11Certificate + 0xf8 [/perago/lib/libpersec.sl]
(16) 0xd3b34a20 getCrl__11CertHandlerFPCcP11Certificate + 0x190 [/perago/lib/libpersec.sl]
(17) 0xd3b52280 updateCrl__5StoreFv + 0x108 [/perago/lib/libpersec.sl]
(18) 0xd3b51944 getPrivKey__5StoreF5DNameb + 0xe4 [/perago/lib/libpersec.sl]
(19) 0xd3b4cc7c signHash__4SignFP6MemBufR5DNameT1 + 0x6c [/perago/lib/libpersec.sl]
(20) 0xd3b27138 sign__7XmlSignFP6MemBufPcR5DNameT1 + 0x1b8 [/perago/lib/libpersec.sl]
(21) 0xd394df20 Java_com_perago_security_Crypto_signXml + 0x380 [/perago/lib/libpersecj.sl]
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (11) at pc=d36f57f8, pid=21566, tid=48
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0.03 jinteg:02.13.06-15:51 PA2.0 (aCC_AP) mixed mode)
# Problematic frame:
# C [libnnz10.sl+0x1f57f8] BN_set_word+0x10
#
# An error report file with more information is saved as hs_err_pid21566.log
#
# Please report this error to HP customer support.
#
run.sh[174]: 21566 Abort(coredump)

The first few lines of hs_err_pid21566.log are as follows:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (11) at pc=d36f57f8, pid=21566, tid=48
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0.03 jinteg:02.13.06-15:51 PA2.0 (aCC
_AP) mixed mode)
# Problematic frame:
# C [libnnz10.sl+0x1f57f8] BN_set_word+0x10
#

--------------- T H R E A D ---------------

Current thread (006e00c8): JavaThread "JMS SessionPool Worker-1" daemon [_threa
d_in_native, id=48, lwp_id=211441]

siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=4cd9c29c

Registers:
r1: 09d7f800 rp: d3732a83 r3: 03ab7988 r4: 00000000
r5: 4cd9c26c r6: 4cd9c280 r7: 4cd9c294 r8: 03a8b700
r9: 4cd9c2d0 r10: 4cd9c2bc r11: 492e4944 r12: 03a8b6db
r17: 00000000 r18: 75c11898 r19: 75ae0d3c r20: 00000000
r21: 4f206580 r22: 00000000 r23: 00000000 r24: 00000000
r25: 03a8b6dd r26: 4cd9c294 dp: 00012c90 ret0: 0ea2db70
ret1: 0ea2db74 sp: 4f2065c0 r31: 4cd9c294
Stack pointer: (sp=4f2065c0)
4f2065a0: 75ae0d3c 00000100 03ab7988 00000000
4f2065b0: 4cd9c294 00000000 4cd9c26c 4cd9c280
Instructions: (pc=d36f57f8)
d36f57d8: e840d000 4fc33f01 e81f1ff7 00000000
d36f57e8: 6bc23fd9 37de0080 375f0000 6bd33fc1
d36f57f8: 4b590010 8f226018 f3170c00 37fc0000

Stack: [4f180000,4f200000), sp=4f2065c0, free space=537k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libnnz10.sl+0x1f57f8] BN_set_word+0x10


libnnz10.sl is an Oracle library.

Any ideas where I can start looking for the cause please? I have tried all options regarding ordering of the SHLIB_PATH and LD_LIBRARY_PATH variables. Even tried renaming libnnz10.sl ;-) - Oracle was not happy and crashed on me.

Thanks

Carlo

[Dennis Handly]

>libnnz10.sl is an Oracle library.

The problem could be there or bad input from the caller or all the way back to #8 in libpersecj.sl. Or back to your java program.

>Any ideas where I can start looking for the cause please? I have tried all options regarding ordering of the SHLIB_PATH and LD_LIBRARY_PATH variables.

Why would the *PATHs have anything to do with it?

One issue could be a thread stack overflow, since you are near the beginning of BN_set_word.

You could figure this out by disassembling the first few words of BN_set_word.
Either from this:
Instructions: (pc=d36f57f8)
d36f57e8: 6bc23fd9 37de0080 375f0000 6bd33fc1
d36f57f8: 4b590010 8f226018 f3170c00 37fc0000

Or by using gdb:
$ gdb /u01/app/oracle/10g/lib32/libnnz10.sl
(gdb) disas BN_set_word BN_set_word+4*8
(gdb) q

If the load/stores are relative to the input args, it is a bad address. If relative to $r30 or $sp, then it is a thread stack overflow.

[Carlo Henrico_1]

Hello Dennis

I am not sure what to make of the reply and gdb's response. gdb is not installed by the customer but mine at my office gives:

(gdb) disas BN_set_word BN_set_word+4*8
Dump of assembler code from 0x1f5288 to 0x1f52a8:
0x1f5288 : stw %rp,-0x14(%sp)
0x1f528c : ldo 0x40(%sp),%sp
0x1f5290 : copy %r26,%r31
0x1f5294 : stw %r19,-0x20(%sp)
0x1f5298 : ldw 8(%r26),%r25
0x1f529c : cmpib,> 1,%r25,0x1f52b0
0x1f52a0 : depd %r23,31,32,%r24
0x1f52a4 : copy %r31,%ret0
0x1f52a8 : cmpb,<>,n %ret0,%r0,0x1f52d8
End of assembler dump.

As far as the paths are concerned, on this site someone indicated that Oracle has a problem in this file and one should re-arrange the LD_LIB/SHLIB search sequence so that it might pick up the correct library first instead of Oracle's.

[Dennis Handly]

>gdb is not installed by the customer but mine at my office gives:
0x1f5298 : ldw 8(%r26),%r25

This indicates that the R26 (first arg) value is out of bounds, and matches the value in si_addr.

>someone indicated that Oracle has a problem in this file and one should re-arrange the LD_LIB/SHLIB search sequence so that it might pick up the correct library first

Do you know where the correct lib is?

[Carlo Henrico_1]

Unfortunately not. Have no idea what is causing this crash.

Carlo

[Dennis Handly]

>Have no idea what is causing this crash.

On my 11.23 IPF system I found those functions in:
/opt/openssl/lib/hpux32/libcrypto.so

I also have a man page and header so I can be dangerous. :-)

It appears that the BIGNUM* passed to BN_set_word is out of bounds.
This could be due to any of the three BIGNUM*
passed to BN_mod_inverse from libpersecj.sl.

Analysis of the core file may indicate that.