HPUX 11.23 STL core dump C_make_facet

[HP UX 11.23 STL core du]

Hi -

We have 2 different HPUX 11.23 64 bit Itanium products, one built with aC+ 06.05 and other with 06.13. Both work perfectly in their standalone mode. but when

product1 tries to load product2 shared library everything goes wrong. application terminated unexpectedly with core -

#0 0xc00000000031eb70:0 in kill+0x30 ()
#0 0xc00000000031eb70:0 in kill+0x30 ()
#1 0xc0000000002432d0:0 in raise+0x30 ()
#2 0xc0000000002e0270:0 in abort+0x190 ()
#3 0xc0000000018b40f0:0 in std::terminate()+0x50 ()
#4 0xc0000000018b2750:0 in __cxa_personality_routine+0xa00 ()
#5 0xc000000001a8e780:0 in _Unwind_RaiseException+0x260 ()
#6 0xc0000000018bb010:0 in __cxa_throw+0x130 ()
#7 0xc00000000175e220:0 in std::string::_C_unlink()+0x220 ()
#8 0xc000000001762f70:0 in std::string::~string()+0x30 ()
#9 0xc0000000017a12b0:0 in std::numpunct::_C_initfacet(std::locale const&)+0x350 ()
#10 0xc00000000176e990:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x1d0 ()
#11 0xc00000000176dae0:0 in std::locale::_C_make_facet(std::locale::id const&,bool,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const


we have applied exact patches on 11.23 machine as per "Available in December 2006 as Patch PHSS_35752" table

http://docs.hp.com/en/11920/HP%20aC++%20Online%20Programmer's%20Guide/faq.htm

But problem still persists.


flags we used to build product2 are -
-D_RWSTD_MULTI_THREAD -D_REENTRANT -D_THREAD_SAFE +DSitanium2 -z -mt -AA +DD64


We are wondering if there are any aC++ compatibility issues?

Thanks

- Ajit

[Dennis Handly]

>application terminated unexpectedly with core

Were there any messages to stderr before the abort?
What are the last few entries on the stack trace?

>flags we used to build product2 are -
-D_RWSTD_MULTI_THREAD -D_REENTRANT -D_THREAD_SAFE +DSitanium2 -z -mt -AA +DD64

If you are linking with libpthread, you must compile ALL aC++ sources with -mt, even product1.

What aC++ runtime do you have? PHSS_36343 is the latest.

[HP UX 11.23 STL core du]

Thanks Dennis -

Last few STL entries are -

std::ostringstream keyParam, formParam;
keyParam << "UserStr" << j; ( fails here )

We are using "-mt" flag for both of our products.

aC++ Runtime (IA: A.06.10, PA: A.03.70)

We tried resetting locale LC_ALL="" but none of that worked.

- Ajit

[HP UX 11.23 STL core du]

aCC Runtime we have on our machine is -
PHSS_34441 aC++ Runtime (IA: A.06.10, PA: A.03.70)

- Ajit

[Dennis Handly]

>Last few STL entries are -

I wanted to see the last entries on the stack trace. Typically the "missing -mt" error occurs at the start.

>We are using "-mt" flag for both of our products.

Are you sure about any third party libs you may be using?

[HP UX 11.23 STL core du]

Thanks Dennis -

Last few entries are as follows -
#8 0xc000000001762f70:0 in std::string::~string()+0x30 ()
#9 0xc0000000017a12b0:0 in std::numpunct::_C_initfacet(std::locale const&)+0x350 ()
#10 0xc00000000176e990:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x1d0 ()
#11 0xc00000000176dae0:0 in std::locale::_C_make_facet(std::locale::id const&,bool,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const
#12 0xc00000004e1da090:0 in SivTech::SystemInit::UpgradeMgr::compareVersions ()
#13 0xc00000004e1d7fc0:0 in SivTech::SystemInit::UpgradeMgr::sortFiles ()

function compareVersion() calls into STL and fails.

We narrowed down our test case where it passes if we skip STL calls, but as soon as we add any stl code it starts failing.

I will be reviewing all our third party libraries for -mt flag

Thanks Again

- Ajit

[Dennis Handly]

#12 0xc00000004e1da090:0 in SivTech::SystemInit::UpgradeMgr::compareVersions
#13 0xc00000004e1d7fc0:0 in SivTech::SystemInit::UpgradeMgr::sortFiles

I wanted at the end of the stack, either in main or a thread.

You might want to recompile compareVersions with +d so you don't have inlining so you can see what is being called at each level.

Frame #11 seems to be in a different shlib than #12.

>I will be reviewing all our third party libraries for -mt flag

You would only need to scan aC++ libs. If compiled with a recent version, you can use footprints(1) to see the -mt state.
Otherwise use "elfdump -dc".

You may want to download a newer gdb so the stack trace as line info?

>We tried resetting locale LC_ALL="" but none of that worked.

Do you still have LANG set to something?

[HP UX 11.23 STL core du]

Thanks Dennis -

We have some third party libraries which are compiled using gcc. will that make any difference?

LANG="" and LC_ALL=""

I will try to get exact stack trace with "+d"

- Ajit

[HP UX 11.23 STL core du]

Hi Dennis -

Got full stack trace with +d option. We narrowed down our test case to just single library. When that gets called core dump happens

ldd on the library where failure occurs shows.

libstd_v2.so.1 => /usr/lib/hpux64/libstd_v2.so.1
libCsup.so.1 => /usr/lib/hpux64/libCsup.so.1
libpthread.so.1 => /usr/lib/hpux64/libpthread.so.1
libm.so.1 => /usr/lib/hpux64/libm.so.1
libunwind.so.1 => /usr/lib/hpux64/libunwind.so.1
libc.so.1 => /usr/lib/hpux64/libc.so.1
libdl.so.1 => /usr/lib/hpux64/libdl.so.1
libuca.so.1 => /usr/lib/hpux64/libuca.so.1


please see below.
#0 0xc00000000031eb70:0 in kill+0x30 ()
#1 0xc0000000002432d0:0 in raise+0x30 ()
#2 0xc000000019c206e0:0 in sigabrt ()
#3
#4 0xc00000000031eb70:0 in kill+0x30 ()
#5 0xc0000000002432d0:0 in raise+0x30 ()
#6 0xc0000000002e0330:0 in abort+0x250 ()
#7 0x400000000134d640:0 in PmAbortProcess ()
#8 0x4000000000677010:0 in fatalSignalHandler ()
#9
#10 0xc0000000000ced70:0 in pthread_mutex_lock+0x610 ()
#11 0xc000000000339d70:0 in __thread_mutex_lock+0xb0 ()
#12 0xc00000000440b530:0 in _HPMutexWrapper::lock(void*)+0x90 ()
#13 0xc000000004412d60:0 in std::basic_string,std::allocator >::operator=(std::basic_string,std::allocator > const&)+0xa0 ()
#14 0xc00000000444df10:0 in std::numpunct::_C_initfacet(std::locale const&)+0x1d0 ()
#15 0xc00000000441e320:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x120 ()
#16 0xc00000000441d3c0:0 in std::locale::_C_make_facet(std::locale::id const&,bo---Type to continue, or q to quit---
ol,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const +0x3e0 ()
#17 0xc0000000044d77e0:0 in std::num_put > >::do_put(std::ostreambuf_iterator >,std::ios_base&,char,int) const+0x4a0 ()
#18 0xc000000057933300:0 in std::num_put > >::put () at /opt/aCC/include_std/rw/numeral:518
#19 0xc000000057935730:0 in std::basic_ostream >& __rw::__rw_insert,int> ()
#20 0xc00000005792f5b0:0 in std::basic_ostream >::operator<< () at /opt/aCC/include_std/ostream:161
#21 0xc00000005792ef60:0 in p_ct_inputRowNotification () at p_ct.cpp:199
#22 0x40000000010c98e0:0 in

There are no third party dependancies and code is compiled with -mt -lpthread
Please let me know any suggestions I can try out.

Thanks

- Ajit

[Dennis Handly]

>Got full stack trace with +d option.

Can you show it to the end?

>We narrowed down our test case to just single library. When that gets called core dump happens

What would it take to for me to duplicate the problem?

>ldd on the library where failure occurs shows.
>libc.so.1 => /usr/lib/hpux64/libc.so.1

Ideally your lib shouldn't be linked against libc.

#9
#10 0xc0000000000ced70:0 in pthread_mutex_lock+0x610
#11 0xc000000000339d70:0 in __thread_mutex_lock+0xb0
#12 0xc00000000440b530:0 in _HPMutexWrapper::lock(void*)+0x90

This is a typical missing -mt abort.

#13 0xc000000004412d60:0 in std::basic_string::operator=(string const&)+0xa0

A string assignment.

#20 0xc00000005792f5b0:0 in std::basic_ostream::operator<<()

Trying to insert an int.

>Please let me know any suggestions I can try out.

Well the horse answer is that you forgot -mt somewhere.
The zebra answer is there is some type of heap corruption that is destroying the string.
Or you have used tricky linker options like -B symbolic and have broken C++'s ODR rule. Or dlopen without RTLD_GLOBAL.

[HP UX 11.23 STL core du]

Thanks Dennis -

>> Ideally your lib shouldn't be linked against libc

Well, we donot explicitly link libc.

>>Well the horse answer is that you forgot -mt somewhere.

I ran "elfdump -dc" product1 binaries which loads prodict2 shared libraries and I think some of them are missing -mt options.
Just to clarify this. How do you identify wheather binary is -mt enabled or not

Is it "ecom options" section which must say -mt on?

I confirmed all product2 binararies are -mt enabled.

- Ajit

[Dennis Handly]

>we do not explicitly link libc.

It was listed with your "ldd output" above. Unless that was for your whole application, not just the one shlib.

>I ran "elfdump -dc" product1 binaries which loads prodict2 shared libraries and I think some of them are missing -mt options.
>How do you identify whether binary is -mt enabled or not
>Is it "ecom options" section which must say -mt on?

It's complicated. That's why you should be using footprints(1). You see "-mt on" only if you are on A.06.07 or later AND you use -mt. For your A.06.05 objects you can't tell.

>I confirmed all product2 binaries are -mt enabled.

Except the problem could be elsewhere.

[HP UX 11.23 STL core du]

Thanks Dennis -

I matched exact product1 flags to build product2 library but still fails. Interesting thing is even if we define -DRW_NO_STL it fails with same stack trace. I beleive RW_NO_STL flag should disable calling rw_ routines?

http://www.roguewave.com/support/docs/hppdocs/genbd/4-2.html

compile flags now -
+d -AA -mt -z -ext +w +We400 +W392 +W302 -Wc,-ansi_for_scope,on -D_POSIX_C_SOURCE=199506L -DHPUX_VERS=1100 +DD64 +DSitanium2 +Ofltacc +DD64 +DSitanium2 -DRW_NO_STL -DHPUX_IPF

link flags -
-Wl,+s +DD64 +d +DSitanium2 -b -Wl,+s -z â mt


Stack trace -
#10 0xc0000000000ced70:0 in pthread_mutex_lock+0x610 ()
#11 0xc000000000339d70:0 in __thread_mutex_lock+0xb0 ()
#12 0xc00000000440b530:0 in _HPMutexWrapper::lock(void*)+0x90 ()
#13 0xc000000004412d60:0 in std::basic_string,std::allocator >::operator=(std::basic_string,std::allocator > const&)+0xa0 ()
#14 0xc00000000444df10:0 in std::numpunct::_C_initfacet(std::locale const&)+0x1d0 ()
#15 0xc00000000441e320:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x120 ()
#16 0xc00000000441d3c0:0 in std::locale::_C_make_facet(std::locale::id const&,bo---Type to continue, or q to quit---
ol,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const
+0x3e0 ()
#17 0xc0000000044d77e0:0 in std::num_put > >::do_put(std::ostreambuf_iterator >,std::ios_base&,char,int) const+0x4a0 ()
#18 0xc000000057406ea0:0 in std::num_put > >::put () at /opt/aCC/include_std/rw/numeral:518
#19 0xc0000000574092d0:0 in std::basic_ostream >& __rw::__rw_insert,int> ()
at /opt/aCC/include_std/ostream.cc:51
#20 0xc000000057403150:0 in std::basic_ostream >::operator<< () at /opt/aCC/include_std/ostream:161

[Dennis Handly]

>Interesting thing is even if we define -DRW_NO_STL it fails with same stack trace. I believe RW_NO_STL flag should disable calling rw_ routines?

You should absolutely NOT use -DRW_NO_STL. This is NOT supported with HP's version of tools.h++. (But you probably aren't using librwtool*.)

>+We400 +W392 +W302

These are old aCC5 messages.

>-Wc,-ansi_for_scope,on

Part of -AA.

>Stack trace

Can you get a stack trace to go all the way to the end?

[Dennis Handly]

I typed up a small example to see the code flow.
You might try printing the data in _C_initfacet:
(gdb) rbreak numpunc.*_C_initfacet
(gdb) p ((std::numpunct*)$r32)[0]
I get:
$1 = {<:locale::facet> = {<__rw::__rw_facet_base> = {<__rw::__rw_synchronize
d> = {_C_mutex = {<__rw::__rw_mutex_base> = {_C_mutex = {
pmutex = 0x40023ea0}}, }}, _C_category = 128,
_C_flags = 0,
_C_ref_count = 0, __vfp = 0x76ded160}, }, <__rw::__rw_nump
unct_impl> = {<__rw::__rw_numpunct_impl_data> = {<__rw::__rw_punct_d
ata> = {_C_dp = 46 '.', _C_ts = 44 ',', _C_gr = {
_C_data = 0x76df5ab0 ""}}, _C_tn = {_C_data = 0x40026a50 "true"},
_C_fn = {_C_data = 0x40026b00 "false"}, tf_defs_ = {{_C_name = 0x0,
_C_indx = 0}, {_C_name = 0x0, _C_indx = 0}}, tf_map_ = {
_C_numdefs = 0, _C_defs = 0x0}}, }, }

You may have to compile with -g0 and have to add this to your source to get debug info for this class:
#include
struct foo : std::numpunct {
int member;
};
void dummy() {
foo *XX = 0;
XX->member = 99;
}

Or compile with -U_HP_INSTANTIATE_T_IN_LIB and you can just debug that function normally: p *this

[Dennis Handly]

It turns out one of your shlibs wasn't compiled with -mt: libpmicuio.sl.32

[HP UX 11.23 STL core du]

Dennis -

Issue is now fixed, After compiling ICU library.

Just wondering what tool we shall use to check -mt flag on libraries compiled
with A.06.05. Is it footprint?

Thanks for all your help.

- Ajit

[HP UX 11.23 STL core du]

Hi Dennis -

Just quick on footprints command on binaries compiler with 06.05. I cann't see -mt option enabled on them?

How to check -mt for older compilers?

- Ajit

[Dennis Handly]

>Just wondering what tool we shall use to check -mt flag on libraries compiled with A.06.05. Is it footprints?

Unfortunately footprints only works if you are on at least A.06.07 or you happen to compile with -g (fluke).

>Just quick on footprints command on binaries compiler with 06.05. I can't see -mt option enabled on them?

That's correct. :-( I finally fixed it.

>How to check -mt for older compilers?

You have to use advanced AI technology to look at the symbols being using and lots of guessing.