Java application dumps on I64

[Mario Dhaenens]

Hi,

I have a java application that is running as a detached process on a I64 server.

Every day it dumps with the following stack trace.
Does any know what the cause is of this dump?

OpenVMS stack trace:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# %SYSTEM-F-ACCVIO (0xc) at pc=9007FC90, pid=545342780, tid=123605632
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0-3 05/09/2008-09:09 IA64 mixed mode)
# Problematic frame:
# C [LIBRTL+0xffffd7d1]
#
%CXXL-F-TERMINATING, terminate() or unexpected() called
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
LIBOTS 0 0000000000012A00 FFFFFFFF90262A00
TRACEBACK - Exception occurred during traceback processing
CXXL$LANGRTL 0 0000000000040D62 FFFFFFFF910DED62
0 FFFFFFFF803F67E2 FFFFFFFF803F67E2
----- Above condition handler called with exception 0000000C
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=0000000000000000, PC=FFFFFFFF90262A00, PS=0000001B
----- End of Exception message
0 FFFFFFFF803E9BC2 FFFFFFFF803E9BC2
LIBOTS 0 0000000000012A00 FFFFFFFF90262A00
TRACEBACK - Exception occurred during traceback processing
CXXL$LANGRTL 0 0000000000040D62 FFFFFFFF910DED62
0 FFFFFFFF803F67E2 FFFFFFFF803F67E2
----- Above condition handler called with exception 0000000C
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=0000000000000000, PC=FFFFFFFF90262A00, PS=0000001B
----- End of Exception message
0 FFFFFFFF803E9BC2 FFFFFFFF803E9BC2
LIBOTS 0 0000000000012A00 FFFFFFFF90262A00
TRACEBACK - Exception occurred during traceback processing
CXXL$LANGRTL 0 0000000000040D62 FFFFFFFF910DED62
0 FFFFFFFF803F67E2 FFFFFFFF803F67E2
----- Above condition handler called with exception 0000000C
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000790B420, PC=FFFFFFFF9007FC90, PS=0000001B
----- End of Exception message
0 FFFFFFFF803E9BC2 FFFFFFFF803E9BC2
TRACEBACK - Exception occurred during traceback processing
LIBRTL 0 0000000000051C90 FFFFFFFF9007FC90
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000F2972 FFFFFFFF9068E972
TRACEBACK - Exception occurred during traceback processing
CXXL$LANGRTL 0 0000000000042BE2 FFFFFFFF910E0BE2
TRACEBACK - Exception occurred during traceback processing
CXXL$LANGRTL 0 0000000000041022 FFFFFFFF910DF022
0 FFFFFFFF803F67E2 FFFFFFFF803F67E2
----- Above condition handler called with exception 0000000C
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000019E, PC=FFFFFFFF9007FC90, PS=0000001B
----- End of Exception message
0 FFFFFFFF803E9BC2 FFFFFFFF803E9BC2
TRACEBACK - Exception occurred during traceback processing
LIBRTL 0 0000000000051C90 FFFFFFFF9007FC90
TRACEBACK - Exception occurred during traceback processing
LIBRTL 0 0000000000057072 FFFFFFFF90085072
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000F28B2 FFFFFFFF9068E8B2
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 0000000000072572 FFFFFFFF9060E572
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000321D2 FFFFFFFF905CE1D2
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000680A2 FFFFFFFF906040A2
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 00000000017C80E2 00000000018300E2
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 000000000109A322 0000000001102322
0 FFFFFFFF803F67E2 FFFFFFFF803F67E2
----- Above condition handler called with exception 0000000C
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000019E, PC=FFFFFFFF9007FC90, PS=0000001B
----- End of Exception message
0 FFFFFFFF803E9BC2 FFFFFFFF803E9BC2
TRACEBACK - Exception occurred during traceback processing
LIBRTL 0 0000000000051C90 FFFFFFFF9007FC90
TRACEBACK - Exception occurred during traceback processing
LIBRTL 0 0000000000057072 FFFFFFFF90085072
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000F28B2 FFFFFFFF9068E8B2
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000A6722 FFFFFFFF90642722
TRACEBACK - Exception occurred during traceback processing
DECC$SHR 0 00000000000D0002 FFFFFFFF9066C002
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 00000000004D4552 000000000053C552
TRACEBACK - Exception occurred during traceback processing
JAVA$NET_SHR 0 0000000000073902 0000000002E1B902
0 000000002F340F41 000000002F340F41
0 000000002F84E4A2 000000002F84E4A2
0 000000002F7B4B02 000000002F7B4B02
0 000000002F843C72 000000002F843C72
0 000000002F7B6C02 000000002F7B6C02
0 000000002F988092 000000002F988092
0 000000002F94A1E2 000000002F94A1E2
0 000000002F905322 000000002F905322
0 000000002F99C1D2 000000002F99C1D2
0 000000002F9A02A2 000000002F9A02A2
0 000000002F3A8752 000000002F3A8752
0 000000002F353DF2 000000002F353DF2
0 000000002F345240 000000002F345240
0 000000002F345A60 000000002F345A60
0 000000002F340620 000000002F340620
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 0000000000A017E2 0000000000A697E2
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 0000000001097F52 00000000010FFF52
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 0000000000A037E2 0000000000A6B7E2
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 0000000000A02882 0000000000A6A882
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 0000000000A02B02 0000000000A6AB02
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 00000000004B5412 000000000051D412
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 00000000016FD242 0000000001765242
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 00000000016FCF42 0000000001764F42
TRACEBACK - Exception occurred during traceback processing
JAVA$HOTSPOT_SHR 0 000000000108B722 00000000010F3722
TRACEBACK - Exception occurred during traceback processing
PTHREAD$RTL 0 00000000000792E2 FFFFFFFF905432E2
TRACEBACK - Exception occurred during traceback processing
PTHREAD$RTL 0 0000000000053942 FFFFFFFF9051D942
TRACEBACK - Exception occurred during traceback processing
PTHREAD$RTL 0 0000000000030492 FFFFFFFF904FA492
0 FFFFFFFF80BA8192 FFFFFFFF80BA8192
DCL 0 000000000006BD22 000000007AE01D22
%TRACE-I-END, end of TRACE stack dump
%CXXL-F-TERMINATING, terminate() or unexpected() called
%CMA-F-EXIT_THREAD, current thread has been requested to exit

Improperly handled condition, image exit forced by last chance handler.
Signal arguments: Number = 0000000000000008
Name = 00000000004081A4
0000000000000000
0000000000000000
0000000000000000
0000000000000000
0000000000000000
FFFFFFFF9051D5E0
000000000000001B

Register dump:
R0 = 0000000000000000 R1 = 000000007B540000 R2 = 0000000000000243
R3 = 000000007B745DF0 R4 = 00000000075DAC68 R5 = 000000007B982020
R6 = 00000000075D9F90 R7 = FFFFFFFFB1768620 R8 = 0000000000000000
R9 = 0000000000000000 R10 = 0000000000000080 R11 = 0000000000000080
SP = 00000000075D9530 TP = 00000000075E1280 R14 = FFFFFFFFB15E7840
R15 = 000000007B745B48 R16 = FFFFFFFF901EBF10 R17 = 0000000000000001
R18 = 0000000000000007 R19 = 0000000000000008 R20 = 000000007AEA1720
R21 = 0000000000000000 R22 = 0000000000000002 R23 = E000000000000050
R24 = 0000000000000004 R25 = 0000000000000006 R26 = FFFFFFFFB15DEEA0
R27 = FFFFFFFFB15DEEDC R28 = 000000000000000F R29 = 00000000075D9560
R30 = 00000000075D93F8 R31 = 0000000000000003 PC = FFFFFFFF9051D5E0
BSP/STORE = 000007FDBE5AAE18 / 000007FDBE5AAE18 PSR = 0000101308026030
IIPA = 0000000000000000
B0 = FFFFFFFF9051D5E0 B6 = FFFFFFFF901EBF10 B7 = FFFFFFFF90526310

Interrupted Frame RSE Backing Store, Size = 19 registers

R32 = 00000000004081A4 R33 = 00000000075E1280 R34 = 0000000000000000
R35 = 00000000004081A4 R36 = 0000000000000004 R37 = 000000007B934000
R38 = 00000000075E1280 R39 = 0000000000000001 R40 = FFFFFFFF9053F210
R41 = C03000000000020A R42 = 000000007B934000 R43 = 00000000004081A4
R44 = 0000000000000000 R45 = 0000000000000000 R46 = 0000000000000000
R47 = 0000000000000000 R48 = 0000000000000000 R49 = FFFFFFFF9051FA30
R50 = C03000000000048F
%CMA-F-EXIT_THREAD, current thread has been requested to exit

Improperly handled condition, image exit forced by last chance handler.
Signal arguments: Number = 0000000000000008
Name = 00000000004081A4
0000000000000000
0000000000000000
0000000000000000
0000000000000000
0000000000000000
FFFFFFFF9051D5E0
000000000000001B

Register dump:
R0 = 0000000000000000 R1 = 000000007B540000 R2 = 0000000000000243
R3 = 000000007B745DF0 R4 = 00000000075DAC68 R5 = 000000007B982020
R6 = 00000000075D9F90 R7 = FFFFFFFFB1768620 R8 = 0000000000000000
R9 = 0000000000000000 R10 = 0000000000000080 R11 = 0000000000000080
SP = 00000000075D8840 TP = 00000000075E1280 R14 = FFFFFFFFB15E7840
R15 = 000000007B745B48 R16 = FFFFFFFF901EBF10 R17 = 0000000000000001
R18 = 0000000000000007 R19 = 0000000000000008 R20 = 00000000800E2000
R21 = 0000000000000000 R22 = 0000000000000002 R23 = E000000000000050
R24 = 0000000000000004 R25 = 0000000000000006 R26 = FFFFFFFFB15DEEA0
R27 = FFFFFFFFB15DEEDC R28 = 000000000000000F R29 = 00000000075D8870
R30 = 000007FDBFFCC020 R31 = 0000000000000003 PC = FFFFFFFF9051D5E0
BSP/STORE = 000007FDBE5AB448 / 000007FDBE5AB448 PSR = 0000101308026030
IIPA = 0000000000000000
B0 = FFFFFFFF9051D5E0 B6 = FFFFFFFF901EBF10 B7 = FFFFFFFF90526310

Interrupted Frame RSE Backing Store, Size = 19 registers

R32 = 00000000004081A4 R33 = 00000000075E1280 R34 = 0000000000000000
R35 = 00000000004081A4 R36 = 0000000000000004 R37 = 000000007B934000
R38 = 00000000075E1280 R39 = 0000000000000001 R40 = FFFFFFFF9053F210
R41 = C03000000000020A R42 = 000000007B934000 R43 = 00000000004081A4
R44 = 0000000000000000 R45 = 0000000000000000 R46 = 0000000000000000
R47 = 0000000000000000 R48 = 0000000000000000 R49 = FFFFFFFF9051FA30
R50 = C03000000000048F
4-JUL-2009 09:30:48
PVECMGR job terminated at 4-JUL-2009 09:30:48.50

Accounting information:
Buffered I/O count: 236286 Peak working set size: 293936
Direct I/O count: 4833 Peak virtual size: 882208
Page faults: 81926 Mounted volumes: 0
Charged CPU time: 0 00:01:15.35 Elapsed time: 0 11:15:33.15

Regards,

/Toine

[Volker Halle]

Toine,

while this answer may not help you much:

The reason for this image termination is an ACCVIO (access violation) by the instruction at PC 9007FC90. The instruction at this PC tried to access an address 0x0000019E, which is an inaccessible address.

You could create a process dump, if you add a SET PROC/DUMP before invoking the program or use a RUN/DUMP to start the detached process.

You can then ANAL/PROC imagename.DMP and find out the exact failing instruction, but it will either be in code owned by HP. By looking around to where the various registers point to, you may be able to spot some ASCII (readable) data and guess what the process was doing at that time.

The peak working set and the peak virtual size may also indicate, that the process ran out of virtual memory, so watch the process during execution.

The process address space or stack seems to also have been sufficiently clobbered, as it also occured lots of exceptions during traceback handling.

Volker.