Tru64 core dump libc - locking error on exception resources shared among threads

[Stephen Dove]

Has anyone seen these errors. This problem occurs intermittently. Is there a way to get more out of the core file?
Here is some information:

OSF1 V5.1 2650 alpha server

We have seen these errors in log files:
exception system: exiting due to multiple internal errors:

locking error on exception resources shared among threads

locking error on exception resources shared among threads

The gdp binary is an ADA application:
file gdp
gdp: COFF format alpha dynamically linked, demand paged executable or object module not stripped - version 3.13-14


Info from the core file:

ladebug gdp core
Welcome to the Ladebug Debugger Version 69 (built May 4 2003 for Tru64 UNIX)
------------------
object file name: gdp
core file name: core
Reading symbolic information ...done
Core file produced from executable 'gdp'
Thread terminated at PC 0x3ff800edbc8 by signal ABRT
(ladebug) where
>0 0x3ff800edbc8 in __kill(...) in applibs/libc.so
#1 0x3ff80221290 in UnknownProcedure18FromFile10(...) in applibs/libc.so
#2 0x3ff80140994 in __tis_raise(...) in applibs/libc.so
#3 0x3ff801d3208 in abort(...) in applibs/libc.so
#4 0x3ff806322b0 in __exc_raise_status_exception(...) in applibs/libexc.so
#5 0x3ff801d56e4 in __exc_remote_lookup_function_entry(...) in applibs/libc.so
#6 0x3ff801d38b8 in __exc_remote_virtual_unwind(...) in applibs/libc.so
#7 0x3ff801d48a8 in __exc_virtual_unwind(...) in applibs/libc.so
#8 0x3ff80632450 in exc_raise_exception(...) in applibs/libexc.so
#9 0x3ff806322e4 in __exc_raise_status_exception(...) in applibs/libexc.so
#10 0x3ff801d5780 in __exc_remote_lookup_function_entry(...) in applibs/libc.so
#11 0x120e21e1c in _A_+do_raise_exception.13S12.ex_raise() in gdp
#12 0x120e1d344 in _A_+raise_interrupt.360S12.ex_utils() in gdp

(ladebug) up 12

(ladebug) wi
void _A_+raise_interrupt.360S12.ex_utils(void): /apex/bango/a/625/install/new_self_build/usr_lib_lib/ex_utils_b.a
[0x120e1d334] stq sp, 0(r28)
[0x120e1d338] bis r31, sp, r18
[0x120e1d33c] bis r31, r16, r9
[0x120e1d340] ldq r27, -9016(gp)
[0x120e1d344] jsr r26, (r27), _A_+raise_interrupt.360S12.ex_utils
*[0x120e1d348] ldah gp, 8006(r26)
[0x120e1d34c] lda gp, -28856(gp)
[0x120e1d350] bis r31, r9, r16
[0x120e1d354] bis r31, sp, r17
[0x120e1d358] ldq r27, -9888(gp)

(ladebug) printregs
$r0 [$v0] = 0 $r1 [$t0] = 0
$r2 [$t1] = 1 $r3 [$t2] = 0
$r4 [$t3] = 1 $r5 [$t4] = 4396973508600
$r6 [$t5] = 8192 $r7 [$t6] = 0
$r8 [$t7] = 0 $r9 [$s0] = 5369868776
$r10 [$s1] = 54 $r11 [$s2] = 5369868776
$r12 [$s3] = 5369982096 $r13 [$s4] = 5423138056
$r14 [$s5] = 5423074112 $r15 [$s6] = 5423138648
$r16 [$a0] = 384599 $r17 [$a1] = 6
$r18 [$a2] = 33554432 $r19 [$a3] = 0
$r20 [$a4] = 0 $r21 [$a5] = 0
$r22 [$t8] = 0 $r23 [$t9] = 4395899907208
$r24 [$t10] = 0 $r25 [$t11] = 0
$r26 [$ra] = 4846637896 $r27 [$t12] = 7598814437094465594
$r28 [$at] = 8028075836850796645 $r29 [$gp] = 4396973421776
$r30 [$sp] = 5423137312 $r31 [$zero]= 0
$f0 = -4.5767114681873503e-246 $f1 = 128
$f2 = 0 $f3 = 0
$f4 = 0 $f5 = 0
$f6 = 0 $f7 = 0
$f8 = 0 $f9 = 0
$f10 = 1.7097077964294047e-306 $f11 = 60402991000000
$f12 = 60402991 $f13 = 1000
$f14 = 1000000000 $f15 = 60402.991000000002
$f16 = 10 $f17 = 0
$f18 = 0.5 $f19 = 0.5
$f20 = 10 $f21 = 1
$f22 = 0 $f23 = 1000000000
$f24 = 43200 $f25 = 43200
$f26 = 0 $f27 = 10
$f28 = 0 $f29 = 0
$f30 = 0 $f31 = 0
$pc = 0x120e1d348 $ps = 0x8
$fpcr = 0x8d00000000000000 $uniq = 0x3ffc0086b68
$vfp = 0x1433e8220
(ladebug)

[John Manger]

It'd be helpful to know the precise versions of things : Tru64 (sizer -v), patch kit (dupatch -track -type kit), and Ada itself (setld -i | grep -i ada).

If some of those are 'low' or old that might be a problem.

Output from an odump -Dl of your 'gdp' would show which libs its actually using - applibs/libc.so isn't the standard libc.so.

John M

[Stephen Dove]

Here is some more info as requested. Thanks for the reply.

# sizer -v
HP Tru64 UNIX V5.1B (Rev. 2650);

# dupatch -track -type kit

* Previous session logs saved in session.log.[1-25]

- This dupatch session is logged in /var/adm/patch/log/session.log

Gathering details of relevant patches, this may take a bit of time

Patches installed on the system came from following software kits:
------------------------------------------------------------------

- T64KIT1001073-V51BB27-E-20070109 OSF540
- T64KIT1001098-V51BB27-E-20070130 OSF540
- T64KIT1001167-V51BB27-ES-20070321 OSF540
- T64KIT1001176-V51BB27-E-20070328 OSF540
- T64KIT1001178-V51BB27-E-20070330 OSF540
- T64KIT1001187-V51BB27-E-20070404 OSF540
- T64KIT1001188-V51BB27-ES-20070404 OSF540
- T64KIT1001205-V51BB27-ES-20070427 OSF540
- T64KIT1001259-V51BB27-E-20070717 OSF540
- T64KIT1001268-V51BB27-ES-20070806 OSF540
- T64KIT1001279-V51BB27-E-20070817 OSF540
- T64KIT1001288-V51BB27-E-20070827 OSF540
- T64KIT1001450-V51BB27-E-20080305 OSF540
- T64KIT1001460-V51BB27-ES-20080310 OSF540
- T64KIT1001509-V51BB27-E-20080611 OSF540
- T64KIT1001520-V51BB27-ES-20080808 OSF540
- T64KIT1001622-V51BB28-E-20090617 OSF540
- T64V51BB25AS0004-20040616 OSF540
- T64V51BB26AS0005-20050502 OSF540
- T64V51BB27AS0006-20061208 OSF540
- T64V51BB28AS0007-20090312 OSF540


ada -v .... Version for Ada compiler
VADS Ada Compiler, Copyright 1984, 1996 VADSself for DEC Alpha AXP,
Digital UNIX V4.0 (Rev. 386), Version 6.2 Mon Apr 21 17:30:18 PDT 1997,
6.2.5(f)


** applibs is a directory I transported the .so's to for Ladebug debugging on another server.

# odump -Dl gdp


***LIBRARY LIST SECTION***
Name Time-Stamp CheckSum Flags Version
gdp:
libexc.so Oct 16 15:53:16 2002 0xb0f9a902 0 osf.1
libc.so Dec 5 16:37:03 2006 0xd4cf2394 0 osf.1

# cd /
# find . -name libexc.so
./usr/shlib/libexc.so
./shlib/libexc.so
# find . -name libc.so
./usr/shlib/libc.so
./shlib/libc.so

# ls -l ./usr/shlib/libexc.so
lrwxr-xr-x 1 root system 21 Oct 13 2005 ./usr/shlib/libexc.so -> ../../shlib/libexc.so
# ls -l ./usr/shlib/libc.so
lrwxr-xr-x 1 root system 19 Oct 13 2005 ./usr/shlib/libc.so -> ../../shlib/libc.so