PA-RISC + STL = core dump

[chris_losinger]

(don't mean to x-post, i think this is probably a better place for this Q)

I'm getting a core dump every time my (C++) app hits a certain section of code. The core dump shows this:

(gdb) where
#0  0xc0000000000a5ad0 in pthread_mutex_lock+0x18 () from /lib/pa20_64/libpthread.1
#1  0xc000000000213820 in __thread_mutex_lock+0x80 () from /lib/pa20_64/libc.2
#2  0xc000000000ac333c in HPMutexWrapper::lock+0x34 () from /lib/pa20_64/libstd.2
#3  0xc000000006139ca4 in DF_NCOA::unstring+0x1dc () from /home/chrisl/DMS/dmserver/lib/libdfxncoa-1.0.sl
...

A few printfs narrowed the problem to this bit of code, which appears in that last function (unstring):

                std::string el;

                el += inString[i]; <<---- crash happens here
                words.push_back(el);

It happens the very first time into this function. It's not a data issue, as inString[i] is a perfectly valid character.

This is deep inside a very large and very complex application, but this failure is repeatable, and, I believe is caused by bad compiler/linker settings.

Right now, this only happens on HPUX64 on PA-RISC (11.23). compiler A.03.90. t works on Windows, many flavors of Linux, Unix including HPUX-i64. It's just HPUX64/PA-RISC that's giving me the problem.

And, I recently solved an identical problem with this code on HPUX-i64 (crash @ pthread_mutex_lock following a call into the STL) by adding the "-mt" flag to my compiler settings. But, adding that flag here doesn't fix anything. It still crashes in the same place, every time.

Any ideas?

[Dennis Handly]

Have you consistently compiled everything with -mt, including all libs?

The abort could have occurred due to one object not compiled with -mt.

Or your string could be corrupted or never initialized.

What is inString[i], a char array element?

[chris_losinger]

i doubt all the modules in the app are built with -mt. it's a gigantic app, with hundreds of modules. but mine is the only one that seems to suffer from this problem. relatively few of the modules in the app use STL, so it's possible we've just been lucky (for many many years...?).

is it a strict requirement that every shared lib in an app be built with -mt ? all or nothing ?

also, the module i'm working on does not make direct use of threads itself, though it may be called from multiple threads in the overapp application.

What is inString[i], a char array element?

yes. inString is a const char *. when the crash happens, inString is 10 chars long, with valid data, and i is 0.

[Dennis Handly]

> so it's possible we've just been lucky (for many many years?).

Yes.  Any any recent aC++ runtime patches?

Or unless heap corruption.

> is it a strict requirement that every shared lib in an app be built with -mt? all or nothing?

Yes.  Except for non-aC++ sources.

> the module I'm working on does not make direct use of threads itself

That doesn't matter, you don't even have to create threads, just the presence of libpthread.sl.

> inString is a const char*.

So the only bad string is el.  Is it really defined right before you append to it?

If so, change to: std::string el(inString[i]);

[chris_losinger]

Any any recent aC++ runtime patches?

I don't know.

Yes.  Except for non-aC++ sources.

Would have to include any and all 3rd party C++ libs which might get used, too ?

So the only bad string is el.  Is it really defined right before you append to it?

Yes. It's declared, i assign a single char to it, then it gets pushed into the vector. Later, it gets more chars appended to it (it's part of a string tokenizing function). But that string isn't the issue - that's just where the problem in manifesting right now. If i remove the one call to that function, i eventually hit another, similar, crash:

(gdb) where
#0  0xc0000000000a5ad0 in pthread_mutex_lock+0x18 () from /lib/pa20_64/libpthread.1
#1  0xc000000000213820 in __thread_mutex_lock+0x80 () from /lib/pa20_64/libc.2
#2  0xc000000000ac333c in HPMutexWrapper::lock+0x34 () from /lib/pa20_64/libstd.2
#3  0xc000000006d71a8c in DF_::CCompStrW::replaceWordlist+0x14c ()

This time, the crash comes inside a map::find (called from replaceWordList).

Is there a way to disable this mutex stuff ? I don't need thread safety around STL access - there is nothing here that needs to be protected - no non-const globals, no  statics, no thread-local data, etc..

[Dennis Handly]

>Is there a way to disable this mutex stuff?

No.  The mutex is to protect the reference count, it has nothing to do with your data.

In fact it may be the reference count for the null string.

You might be able to get somewhere by dumping raw data:

(gdb) frame 3

(gdb) info reg

(gdb) disas $pc-4*20 $pc+4*8

[chris_losinger]

that gives me a lot of ... information (?)

Dump of assembler code from 0xc0000000000a5a80 to 0xc0000000000a5af0:
0xc0000000000a5a80 <pthread_mutex_trylock+0x3630>:      ldw 0x50(%r19),%r26
0xc0000000000a5a84 <pthread_mutex_trylock+0x3634>:      ldi 0,%r25
0xc0000000000a5a88 <pthread_mutex_trylock+0x3638>:      call 0xc0000000000ca9f8 <_lwp_kill>
0xc0000000000a5a8c <pthread_mutex_trylock+0x363c>:      ldo -0x30(%sp),%ret1
0xc0000000000a5a90 <pthread_mutex_trylock+0x3640>:      ldd -0x468(%sp),%dp
0xc0000000000a5a94 <pthread_mutex_trylock+0x3644>:      ldw -0x45c(%sp),%r7
0xc0000000000a5a98 <pthread_mutex_trylock+0x3648>:      copy %r7,%ret0
0xc0000000000a5a9c <pthread_mutex_trylock+0x364c>:      ldd -0x4a0(%sp),%rp
0xc0000000000a5aa0 <pthread_mutex_trylock+0x3650>:      ldd -0x470(%sp),%r7
0xc0000000000a5aa4 <pthread_mutex_trylock+0x3654>:      ldd -0x478(%sp),%r6
0xc0000000000a5aa8 <pthread_mutex_trylock+0x3658>:      ldd -0x480(%sp),%r5
0xc0000000000a5aac <pthread_mutex_trylock+0x365c>:      ldd -0x488(%sp),%r4
0xc0000000000a5ab0 <pthread_mutex_trylock+0x3660>:      ret
0xc0000000000a5ab4 <pthread_mutex_trylock+0x3664>:      ldd,mb -0x490(%sp),%r3
0xc0000000000a5ab8 <pthread_mutex_lock>:        std %rp,-0x10(%sp)
0xc0000000000a5abc <pthread_mutex_lock+0x4>:    ldo 0x90(%sp),%sp
0xc0000000000a5ac0 <pthread_mutex_lock+0x8>:    mfctl %tr3,%r31
0xc0000000000a5ac4 <pthread_mutex_lock+0xc>:    std %dp,-0x90(%sp)
0xc0000000000a5ac8 <pthread_mutex_lock+0x10>:   cmpb,*= %r0,%r26,0xc0000000000a5cfc <pthread_mutex_lock+0x244>
0xc0000000000a5acc <pthread_mutex_lock+0x14>:   std %r26,-0x40(%ret1)
0xc0000000000a5ad0 <pthread_mutex_lock+0x18>:   ldh 0(%r26),%r25
0xc0000000000a5ad4 <pthread_mutex_lock+0x1c>:   ldi 0x36,%r24
0xc0000000000a5ad8 <pthread_mutex_lock+0x20>:   cmpb,<>,n %r24,%r25,0xc0000000000a5cfc <pthread_mutex_lock+0x244>
0xc0000000000a5adc <pthread_mutex_lock+0x24>:   ldw 4(%r26),%r23
0xc0000000000a5ae0 <pthread_mutex_lock+0x28>:   cmpb,ev %r23,%r0,0xc0000000000a5b0c <pthread_mutex_lock+0x54>
0xc0000000000a5ae4 <pthread_mutex_lock+0x2c>:   stw %r23,-0x7c(%sp)
0xc0000000000a5ae8 <pthread_mutex_lock+0x30>:   addil L'-0x800,%dp,%r1
0xc0000000000a5aec <pthread_mutex_lock+0x34>:   ldd 0x690(%r1),%r20
End of assembler dump.

 i have no idea what that tells me, though.

[Dennis Handly]

> that gives me a lot of ... information

I only see the output of the last gdb command?

The only thing useful is:

<pthread_mutex_lock+0x18>:   ldh 0(%r26),%r25

So after doing this, please redo the previous 3 commands.