HPUX 11.23 STL core dump C_make_facet

[HP UX 11.23 STL core du]

Hi -

We have 2 different HPUX 11.23 64 bit Itanium products, one built with aC+ 06.05 and other with 06.13. Both work perfectly in their standalone mode. but when

product1 tries to load product2 shared library everything goes wrong. application terminated unexpectedly with core -

#0 0xc00000000031eb70:0 in kill+0x30 ()
#0 0xc00000000031eb70:0 in kill+0x30 ()
#1 0xc0000000002432d0:0 in raise+0x30 ()
#2 0xc0000000002e0270:0 in abort+0x190 ()
#3 0xc0000000018b40f0:0 in std::terminate()+0x50 ()
#4 0xc0000000018b2750:0 in __cxa_personality_routine+0xa00 ()
#5 0xc000000001a8e780:0 in _Unwind_RaiseException+0x260 ()
#6 0xc0000000018bb010:0 in __cxa_throw+0x130 ()
#7 0xc00000000175e220:0 in std::string::_C_unlink()+0x220 ()
#8 0xc000000001762f70:0 in std::string::~string()+0x30 ()
#9 0xc0000000017a12b0:0 in std::numpunct::_C_initfacet(std::locale const&)+0x350 ()
#10 0xc00000000176e990:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x1d0 ()
#11 0xc00000000176dae0:0 in std::locale::_C_make_facet(std::locale::id const&,bool,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const


we have applied exact patches on 11.23 machine as per "Available in December 2006 as Patch PHSS_35752" table

http://docs.hp.com/en/11920/HP%20aC++%20Online%20Programmer's%20Guide/faq.htm

But problem still persists.


flags we used to build product2 are -
-D_RWSTD_MULTI_THREAD -D_REENTRANT -D_THREAD_SAFE +DSitanium2 -z -mt -AA +DD64


We are wondering if there are any aC++ compatibility issues?

Thanks

- Ajit

[Dennis Handly]

>application terminated unexpectedly with core

Were there any messages to stderr before the abort?
What are the last few entries on the stack trace?

>flags we used to build product2 are -
-D_RWSTD_MULTI_THREAD -D_REENTRANT -D_THREAD_SAFE +DSitanium2 -z -mt -AA +DD64

If you are linking with libpthread, you must compile ALL aC++ sources with -mt, even product1.

What aC++ runtime do you have? PHSS_36343 is the latest.

[HP UX 11.23 STL core du]

Thanks Dennis -

Last few STL entries are -

std::ostringstream keyParam, formParam;
keyParam << "UserStr" << j; ( fails here )

We are using "-mt" flag for both of our products.

aC++ Runtime (IA: A.06.10, PA: A.03.70)

We tried resetting locale LC_ALL="" but none of that worked.

- Ajit

[HP UX 11.23 STL core du]

aCC Runtime we have on our machine is -
PHSS_34441 aC++ Runtime (IA: A.06.10, PA: A.03.70)

- Ajit

[Dennis Handly]

>Last few STL entries are -

I wanted to see the last entries on the stack trace. Typically the "missing -mt" error occurs at the start.

>We are using "-mt" flag for both of our products.

Are you sure about any third party libs you may be using?

[HP UX 11.23 STL core du]

Thanks Dennis -

Last few entries are as follows -
#8 0xc000000001762f70:0 in std::string::~string()+0x30 ()
#9 0xc0000000017a12b0:0 in std::numpunct::_C_initfacet(std::locale const&)+0x350 ()
#10 0xc00000000176e990:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x1d0 ()
#11 0xc00000000176dae0:0 in std::locale::_C_make_facet(std::locale::id const&,bool,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const
#12 0xc00000004e1da090:0 in SivTech::SystemInit::UpgradeMgr::compareVersions ()
#13 0xc00000004e1d7fc0:0 in SivTech::SystemInit::UpgradeMgr::sortFiles ()

function compareVersion() calls into STL and fails.

We narrowed down our test case where it passes if we skip STL calls, but as soon as we add any stl code it starts failing.

I will be reviewing all our third party libraries for -mt flag

Thanks Again

- Ajit

[Dennis Handly]

#12 0xc00000004e1da090:0 in SivTech::SystemInit::UpgradeMgr::compareVersions
#13 0xc00000004e1d7fc0:0 in SivTech::SystemInit::UpgradeMgr::sortFiles

I wanted at the end of the stack, either in main or a thread.

You might want to recompile compareVersions with +d so you don't have inlining so you can see what is being called at each level.

Frame #11 seems to be in a different shlib than #12.

>I will be reviewing all our third party libraries for -mt flag

You would only need to scan aC++ libs. If compiled with a recent version, you can use footprints(1) to see the -mt state.
Otherwise use "elfdump -dc".

You may want to download a newer gdb so the stack trace as line info?

>We tried resetting locale LC_ALL="" but none of that worked.

Do you still have LANG set to something?

[HP UX 11.23 STL core du]

Thanks Dennis -

We have some third party libraries which are compiled using gcc. will that make any difference?

LANG="" and LC_ALL=""

I will try to get exact stack trace with "+d"

- Ajit

[HP UX 11.23 STL core du]

Hi Dennis -

Got full stack trace with +d option. We narrowed down our test case to just single library. When that gets called core dump happens

ldd on the library where failure occurs shows.

libstd_v2.so.1 => /usr/lib/hpux64/libstd_v2.so.1
libCsup.so.1 => /usr/lib/hpux64/libCsup.so.1
libpthread.so.1 => /usr/lib/hpux64/libpthread.so.1
libm.so.1 => /usr/lib/hpux64/libm.so.1
libunwind.so.1 => /usr/lib/hpux64/libunwind.so.1
libc.so.1 => /usr/lib/hpux64/libc.so.1
libdl.so.1 => /usr/lib/hpux64/libdl.so.1
libuca.so.1 => /usr/lib/hpux64/libuca.so.1


please see below.
#0 0xc00000000031eb70:0 in kill+0x30 ()
#1 0xc0000000002432d0:0 in raise+0x30 ()
#2 0xc000000019c206e0:0 in sigabrt ()
#3
#4 0xc00000000031eb70:0 in kill+0x30 ()
#5 0xc0000000002432d0:0 in raise+0x30 ()
#6 0xc0000000002e0330:0 in abort+0x250 ()
#7 0x400000000134d640:0 in PmAbortProcess ()
#8 0x4000000000677010:0 in fatalSignalHandler ()
#9
#10 0xc0000000000ced70:0 in pthread_mutex_lock+0x610 ()
#11 0xc000000000339d70:0 in __thread_mutex_lock+0xb0 ()
#12 0xc00000000440b530:0 in _HPMutexWrapper::lock(void*)+0x90 ()
#13 0xc000000004412d60:0 in std::basic_string,std::allocator >::operator=(std::basic_string,std::allocator > const&)+0xa0 ()
#14 0xc00000000444df10:0 in std::numpunct::_C_initfacet(std::locale const&)+0x1d0 ()
#15 0xc00000000441e320:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x120 ()
#16 0xc00000000441d3c0:0 in std::locale::_C_make_facet(std::locale::id const&,bo---Type to continue, or q to quit---
ol,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const +0x3e0 ()
#17 0xc0000000044d77e0:0 in std::num_put > >::do_put(std::ostreambuf_iterator >,std::ios_base&,char,int) const+0x4a0 ()
#18 0xc000000057933300:0 in std::num_put > >::put () at /opt/aCC/include_std/rw/numeral:518
#19 0xc000000057935730:0 in std::basic_ostream >& __rw::__rw_insert,int> ()
#20 0xc00000005792f5b0:0 in std::basic_ostream >::operator<< () at /opt/aCC/include_std/ostream:161
#21 0xc00000005792ef60:0 in p_ct_inputRowNotification () at p_ct.cpp:199
#22 0x40000000010c98e0:0 in

There are no third party dependancies and code is compiled with -mt -lpthread
Please let me know any suggestions I can try out.

Thanks

- Ajit

[Dennis Handly]

>Got full stack trace with +d option.

Can you show it to the end?

>We narrowed down our test case to just single library. When that gets called core dump happens

What would it take to for me to duplicate the problem?

>ldd on the library where failure occurs shows.
>libc.so.1 => /usr/lib/hpux64/libc.so.1

Ideally your lib shouldn't be linked against libc.

#9
#10 0xc0000000000ced70:0 in pthread_mutex_lock+0x610
#11 0xc000000000339d70:0 in __thread_mutex_lock+0xb0
#12 0xc00000000440b530:0 in _HPMutexWrapper::lock(void*)+0x90

This is a typical missing -mt abort.

#13 0xc000000004412d60:0 in std::basic_string::operator=(string const&)+0xa0

A string assignment.

#20 0xc00000005792f5b0:0 in std::basic_ostream::operator<<()

Trying to insert an int.

>Please let me know any suggestions I can try out.

Well the horse answer is that you forgot -mt somewhere.
The zebra answer is there is some type of heap corruption that is destroying the string.
Or you have used tricky linker options like -B symbolic and have broken C++'s ODR rule. Or dlopen without RTLD_GLOBAL.