Crashing debugger on Itanium...

[Olivier Hurez-Martin]

Environment:

OVMS IA64 V8.2-1 and V8.2 all patched up.
C compiler V7.1-11

Problem:

Threaded application hangs when run /nodebug.
I need the debugger to look at the problem, but the debugger crashes right away (not executing my code) on some DWARF debug data format problem (I have no INI file for the debugger; DBG$INIT logical is undefined)

I logged a call with HP, but got the traditional answer: please provide sample code that reproduces the problem; and I just can't do that; plus it really looks like a problem either with the debugger or with the compiler...)

Here below is the output of the debug session; would anyone here have encountered a similar issue before or have any idea for a cause/ research directions/ Advice of any kind...?

Thanks in advance,

-- Olivier.



$ orb:==$awm_api_bin:aws_orb
$ orb
%DEBUG-I-INFODWARF, error reading Dwarf info: Section 0a extends outside file
%DEBUG-I-INFODWARF, error reading Dwarf info: Section 0c extends outside file
%DEBUG-I-INFODWARF, error reading Dwarf info: SHT_VMS_FIXUP section 10 size 379cf0 not multiple of 18
%DEBUG-I-INFODWARF, error reading Dwarf info: SHT_VMS_FIXUP section 12 size 37bee0 not multiple of 18

OpenVMS I64 Debug64 Version V8.2-017

%DEBUG-F-ACCVIO, access violation, reason mask=00, virtual address=0000000000000000, PC=0000000000038050, PS=0000001B
%DEBUG-I-INITIAL, Language: C, Module: AWM_ORB
%DEBUG-I-NOTATMAIN, Type GO to reach MAIN program

DBG> sho call
module name routine name line rel PC abs PC
*AWM_ORB __MAIN 0000000000000000 0000000000040000
SHARE$PTHREAD$RTL_CODE2 0000000000049B90 FFFFFFFF84547B90
SHARE$PTHREAD$RTL_CODE2 00000000000006C0 FFFFFFFF844FE6C0
FFFFFFFF80B0E240 FFFFFFFF80B0E240

DBG> pthread thread
Thread Name State Substate Policy Pri
------ ------------------------- --------------- ----------- ------------ ---
1 default thread running V2 SCHED_OTHER 11
-3 exit-handling thread blocked SCHED_FIFO 32

DBG> pthread thread -f
main thread 1 (running) "default thread" (0x000000000021ED90), created by pthread
Scheduling: home RAD 0, current RAD 0, SCHED_OTHER policy at priority 11 q_num: 0, average cpu time: 2.82, average sched
interval: 3.18, last run stamp: 423826.050, last ready stamp: 423826.050, last adj stamp: 423826.050, cpu time >=
423831.491063380, V2, on kid 0x0000000021000992.
PKTA 0x7FFEFF98, VP critical set, (DEBUG suspend; real vp-critical clear), history 0x00000000; next block sequence num 1,
reference count is 2, static_ctx: (sp: 0x00000000, fp_control: 0x0000000000000000, persona: 0000000000000001, asten:
0000000000000001); Last dispatched from T-1, to T-1; governed by VP 2
ASTs are enabled
16 pagefaults: 0x00000000003E4000, 0x0000000000422000, 0x000000000053A000, >0x0000000000166000, 0x00000000005DE000,
0x0000000000518000
Alloc flags: teb-not-stack, def-stack, foreign-stack, separate-tsd, tis-tsd; Self flags: no-resched, debug-susp-others; Debug
flags: activated
Thread specific data: 0=0x000000000021F384
Stack: SP unavailable, base is 0x000000007ACE6000 (size: 986603520 bytes), guard area at 0x0000000040000000 (size: 0 bytes)
General cancelability enabled, asynch cancelability disabled
The thread's start function and argument are 0x00185538 (0x000000007ACE1B40)
The thread's latest errno is 0
exit handler thread -3 (blocked) "exit-handling thread" (0x00000000003AF300), created by pthread
Blocked on indeterminate object
No stack trace available
Scheduling: home RAD 0, current RAD 0, SCHED_FIFO policy at priority 32 q_num: 32, average cpu time: 0.00, average sched
interval: 0.01, last run stamp: 423825.540, last ready stamp: 423825.540, cpu time: 0.000167861
PKTA 0x00000000, VP critical history 0x00000000; next block sequence num 1, reference count is 2, static_ctx: (sp: 0x003AD850,
fp_control: 0x0000000000000000, persona: 0000000000000001, asten: 0000000000000001); Last dispatched from T1, to T1; governed
by global lock
ASTs are enabled
Object flags: daemon; Self flags: no-resched; Sched flags: run-til-block, prio-preempt, fixed-prio, report-preemption
Thread specific data: 0=0x00000000003AF8F4
Stack: SP 0x00000000003AD850, base is 0x00000000003AE000 (size: 49152 bytes), yellow zone at 0x00000000003A2000 (size: 8192
bytes), guard area at 0x00000000003A0000 (size: 8192 bytes)
General cancelability enabled, asynch cancelability disabled
The thread's start function and argument are 0x00243758 (0x0000000000000000)
The thread's handle was stored at 0x00230188.
The thread's latest errno is 0


DBG> sho stack

Invocation block 0 Invocation handle 8786429346552

GP: 3604480
PC: __MAIN
In prologue region
RETURN PC: SHARE$PTHREAD$RTL_CODE2+301968

SP: 2060326896
Is memory stack frame:
previous SP: 2060326896
BSP: 8786429346552
Is register stack frame:
previous BSP: 8786429346464

Has handler:
function value: 2718952
handler data at: 1468400

CFM: 8
No locals Outs R32 : R39


Invocation block 1 Invocation handle 8786429346464

GP: 4390912
PC: SHARE$PTHREAD$RTL_CODE2+301968
RETURN PC: SHARE$PTHREAD$RTL_CODE2+1728

SP: 2060326896
Is memory stack frame:
previous SP: 2060327728
BSP: 8786429346464
Is register stack frame:
previous BSP: 8786429346376

Has handler:
function value: 2363744
handler data at: -2038210032

CFM: 1427
Ins/Locals R32 : R42 Outs R43 : R50

[Volker Halle]

Olivier,

you can also look at the hanging PTHREAD application with SDA:

$ ANAL/SYS
SDA> set proc/id=
SDA> PTHREAD command ! same commands as with DEBUG

Does $ ANAL/IMAGE report any problems ?

The ACCVIO happens at PC=38050, which code is loaded at this address:

SDA> SHOW PROC/IMAGE/ID=

Volker.

[Olivier Hurez-Martin]

Thank you, Volker, for the speedy answer and valuable info!

I tried SDA before (and loaded the .DSF in it, to get some symbols on the SHOW CALLS) but I couldn't get much more out of it... Maybe with your help... :-)

The ANA/IMA and ANA/IMA/DEBUG do not detect any error.

While I was writing the note, I also did another full build - with slightly different link options (linking against CMS$OPEN_RTL.EXE and CMA$OPEN_LIB_SHR.EXE instead of PTHREAD$RTL.EXE) and now the process exits upon an explicit abort() call, instead of hanging, so I cannot use SDA anymore but I got more data out of the stack dump, which I am going to try and analyze (quick review says that it is error handling responding to the unability to fetch some data from a structure that was supposed to be fed much earlier and I'd love to be able to use Debug to find out why...)

Debug however still crashes with that executable, apparently before executing any of my code, and funnily enough at the exact same address than before (quite different from the stack dump PC.)

According to the .MAP file, that 38050 address is right inside a fairly large static *data structure* (!?!) in the BSS area.


$BSS$
[...snip...]

_AWSERVICEWRAPPEREXRESULTS 000378F0 0003D2EF 00005A00 ( 23040.) OCTA 4 OVR,REL,GBL,NOSHR,NOEXE, WRT,NOVEC,NOMOD
000378F0 0003D2EF 00005A00 ( 23040.) OCTA 4


This data structure is declared in the midst of function definitions in a C source:

// ... some definitions
awm_strtrMessage _awServiceWrapperExResults[ 256 ];
// some other definitions ...

where awm_strtrMessage is around 16 longwords, at first sight...

I am going to move that thing up to the beginning of the module: something tells me that it would move the address of the crash...

Thanks for the help!

-- Olivier.

[J Nelson]

>%DEBUG-I-INFODWARF, error reading Dwarf info: Section 0a extends outside file
>%DEBUG-I-INFODWARF, error reading Dwarf info: Section 0c extends outside file
>%DEBUG-I-INFODWARF, error reading Dwarf info: SHT_VMS_FIXUP section 10 size 379cf0 not multiple of 18
>%DEBUG-I-INFODWARF, error reading Dwarf info: SHT_VMS_FIXUP section 12 size 37bee0 not multiple of 18

These messages mean that the image file is damaged. Once these appear, it is not surprising the debugger ACCVIOs and later hangs.

Either the compiler or the linker is broken.

-Jeff

[Volker Halle]

Olivier,

if you are interested in finding out about the %DEBUG-I-INFODWARF messages, I would suggest that you collect a little bit more information about the 'failing image':

- ANAL/IMA/SECT=ALL/SEGMENT=ALL/OUT=file (WARNING: this can create a huge output file)
- how was the image linked (exact LINK commands) and what happened with the image afterwards ?
- any shareable images involved ?
- are the sections 0a,0c,10,12 really debug sections ?
- can you confirm (from the above ANAL/IMA output) the INFODWARF diagnosis ?

After you've collected the above information and you still can't find the error, you should probably re-escalate the call to HP and provide these additional details.

Volker.

[Olivier Hurez-Martin]

Jeff, I suspect a corruption indeed but ANA/IMA does not bring it to light. If I run the same image /nodebug, it looks like better behaved.
Also, I rebuilt many times, on two different boxes, ensured no hidden warnings, eliminated compilation messages down to informational included, checked for disk errors, ... Still to no avail.

The ANA/IMA comes out clean, no matter what additional qualifiers I used with it...


Volker, here is my compilation command (from the MAP file)

LINK/DEBUG/TRACEBACK/SYSEXE=SELECTIVE/THREADS_ENABLE=(UPCALLS,MULTIPLE_KERNEL_THREADS)/MAP=LOC_MAP:/FULL/CROSS/DSF=LOC_BIN:AWSERVICE
S.DSF/EXE=LOC_BIN:AWSERVICES.EXE LOC_OBJ:AWSERVICES.OBJ,LOC_SRC:AWSERVICES.OPT/OPT
<$1$DKC301:[SOURCE.AGENT_TECH.BASICS.AWSERVICES.SRC]AWSERVICES.OPT;6>
awscomm_lib:libawscomm.olb/lib/inc=(util_thrdux,awm_clires,util_access,vmsfunctions)
aws_icf_lib:libicfapi.olb/lib
toolbox_lib:libtoolbox.olb/lib
util_lib:libutil.olb/lib
local_lib:libawlocal.olb/lib
sys$share:cma$open_rtl.exe/share
sys$share:cma$open_lib_shr.exe/share
!sys$share:pthread$rtl.exe/share

And the list of shared images, from the ANA/IMA

Shareable Image List
CMA$TIS_SHR
(LESS/EQUAL, 1., 5.)
DECC$SHR
(LESS/EQUAL, 1., 1.)
LIBRTL
(LESS/EQUAL, 1., 1.)
LIBOTS
(LESS/EQUAL, 1., 3.)
PTHREAD$RTL
(LESS/EQUAL, 1., 5.)
SYS$PUBLIC_VECTORS
CMA$OPEN_RTL
(LESS/EQUAL, 4., 5.)
CMA$OPEN_LIB_SHR
(LESS/EQUAL, 1., 0.)
SYS$BASE_IMAGE
Component List, Image / Current System Version
SYS$K_VERSION_BASE_IMAGE (3.0 / 3.0)



As you can see, nothing really outstanding...

There are 8 sections only (and 7 segments) in the output of ANAL/IMA/SECT=ALL/SEGMENT=ALL. Would that be because I specify .DSF files?

SECTION SUMMARY Table at file address 001BD800, 9. sections, 64. bytes each, 576. total bytes

Number Type Name File Address File Size Flags
0. NULL 0000000000000000 0000000000000000 ------------------------------
1. STRTAB .shstrtab 00000000001BD600 0000000000000056 ------------------------------
2. STRTAB .strtab 00000000001BD400 0000000000000001 ------------------------------
3. VMS_TRACE .debug_line 0000000000125600 000000000008DB76 ------------------------------
4. VMS_TRACE .trace_abbrev 00000000001B3200 000000000000088E ------------------------------
5. VMS_TRACE .trace_info 00000000001B3C00 000000000000848E ------------------------------
6. VMS_TRACE .trace_aranges 00000000001BC200 0000000000000DB0 ------------------------------
7. NOTE .note 00000000001BD200 0000000000000148 ------------------------------
8. STRTAB .dynstr 000000000010DFE0 0000000000000070 -A----------------------------


It looks like the debugger is somehow thrown out by something missing in the DWARF format the linker puts together... Results from Debug and ANA/IMA do not match at all.

I will definitely escalate the call with HP.

Thanks a lot, guys, for the help so far!

-- Olivier.

[Olivier Hurez-Martin]

BTW, nothing happened to the image after it was built. I execute it from the very directory...

[Olivier Hurez-Martin]

Before I try to escalate this, I am going to try link without /DSF and compilation with /DEBUG=SYMBOLS instead of just /DEBUG, as suggested by the support engineer.

[Volker Halle]

Oliver,

you're using a .DSF file (debug symbol table file).

Could you try an ANAL/OBJ LOC_BIN:AWSERVICES.DSF and see, whether it reports any errors ?

Volker.

[Olivier Hurez-Martin]

Actually, although CC /DEBUG=SYMBOLS did not help at all, I also tried linking /DEBUG and without /DSF and that cleared the problem! I can now use the debug utility on the generated image!

The build deleted the old .DSF files so I can't answer Volker's question right now, but I will as soon as I get some more time (basically early next week after I deliver my kit to QA). I will then rebuild with /DEBUG/DSF, verify that the debugger still crashes with that image and check whether ANA/OBJ finds anything wrong with the .DSF file, and report the results here as well as to the support engineer who deals with my case.

Thanks for the suggestion, Volker! I just didn't think of it :-/

Regards,

-- Olivier.