Application crash in exit ()

[FRESNEDA Christine]

Our application is crashing on HPUX 11.11 with the following stack:

# gdb ./MTNM_AM.exe core
Detected 64-bit executable.
Invoking /opt/langtools/bin/gdb64.
HP gdb 5.2 for PA-RISC 2.0 (wide), HP-UX 11.00
and target hppa2.0w-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.2 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `MTNM_AM.exe'.
Program terminated with signal 11, Segmentation fault.
SEGV_UNKNOWN - Unknown Error

warning: Load module /usr/opt/temip/lib/libwtc9.sl has been stripped.
Debugging information is not available.

#0 0xc00000000029b0a4 in .stub+0xc9c () from /lib/pa20_64/libc.2
(gdb) where
#0 0xc00000000029b0a4 in .stub+0xc9c () from /lib/pa20_64/libc.2
#1 0xc00000000029dd18 in free+0x1a8 () from /lib/pa20_64/libc.2
#2 0xc00000000030bfc0 in __thread_mutex_free+0xa0 () from /lib/pa20_64/libc.2
#3 0xc0000000014dc328 in _HPMutexWrapper::~_HPMutexWrapper+0x58 () from /lib/pa20_64/libstd_v2.2
#4 0xc0000000014dccac in __rw::__null_string_ref_rep,std::allocator,__rw::__string_ref_rep<:allocator>>>::~__null_string_ref_rep,std::allocator,__rw::__string_ref_rep<:allocator>>>+0x34 () from /lib/pa20_64/libstd_v2.2
#5 0xc000000001014d78 in __cxxTerm_body+0x48 () from /usr/lib/pa20_64/libCsup.2
#6 0xc000000001014ce0 in __cxxTerm+0x20 () from /usr/lib/pa20_64/libCsup.2
#7 0xc0000000002867f0 in exit+0x88 () from /lib/pa20_64/libc.2
#8 0xc000000000031d04 in UT_exit+0x2c () from /usr/lib/pa20_64/dld.sl
#9 0xc00000000003268c in exit+0x14 () from /usr/lib/pa20_64/dld.sl
#10 0xc00000000000a780 in $START$+0xb0 () from /usr/lib/pa20_64/dld.sl
Error accessing memory address 0x0: Invalid argument.

Any idea?

Is there any recommended patch to be installed?

regards,
Christine

[FRESNEDA Christine]

some additional information:


ux138155# swlist -l fileset|grep thread
# PHCO_27633 1.0 Pthread.h fix and new enhancement
# PHCO_36229 1.0 pthread library cumulative patch
# PHKL_25593 1.0 Debugged thread hang
# PHKL_25652 1.0 thread nostop for NFS, rlimit max value fix
# PHKL_27179 1.0 Corrected reference to thread register state
# PHKL_28122 1.0 signals,threads enhancement,Psets Enablement
# PHKL_32002 1.0 physio thread performance degradation
# PHKL_32005 1.0 thread suspend, DaS, panic, physio




ux138155# chatr MTNM_AM.exe
MTNM_AM.exe:
64-bit ELF executable
shared library dynamic path search:
LD_LIBRARY_PATH enabled first
SHLIB_PATH enabled second
embedded path enabled third /usr/opt/tns/lib:/usr/opt/temip/lib:/usr/opt/temip/CRB520/lib:/usr/opt/temip/orbix/610/shlib/pa20_64:/usr/opt/temip/orbix/610/shlib/default/pa20_64
shared library list:
libtemip_crb_ec_base.sl
libAMLibCRBinit.sl
libxerces-c1_7_0.sl
libtemip_crb_sl.sl
libit_poa_acca0331.5
libit_art_acca0331.5
libit_ifc_acca0331.5
libtfc_sp.sl
libtfc_wp.sl
libtfc_fmk.sl
libtfc_event_filter.sl
libtemip_event_filter.sl
libmcc_parpml.sl
libmcc_exec.sl
libmcc_sys.sl
libtls76-m.sl
libtns.sl
libm.2
libpthread.1
libnsl.1
libstd_v2.2
libCsup_v2.2
libcl.2
libc.2
libdl.1
shared library binding:
deferred
global hash table disabled
global hash table size 1103
shared library mapped private disabled
shared library segment merging disabled
shared vtable support disabled
explicit unloading disabled
linkage table protection disabled
segments:
index type address flags size
6 text 4000000000000000 z---c- D (default)
7 data 8000000100000000 ---m-- D (default)
executable from stack: D (default)
static branch prediction disabled
kernel assisted branch prediction enabled
lazy swap allocation for dynamic segments disabled
nulptr references disabled

[Dennis Handly]

Your application is threaded. Have you compiled everything with -mt?
What version of aC++ do you have? What is your version of the aC++ runtime lib patch?

[FRESNEDA Christine]

Hello,

We have recompiled the executable with option 'mt' but it doesn't make any difference.

I have attached the trace of the build.

Any idea, please?

regards,
Christine

[Dennis Handly]

>but it doesn't make any difference. Any idea, please?

(Note: You don't need -D__HPACC_USING_MULTIPLIES_IN_FUNCTIONAL if you are using -AA.)

Did you make sure that ALL aC++ objects, even if from third party libs are compiled with -mt?

You never mentioned what version of aC++ and the runtime you have?

One other issue would be the use of -Bsymbolic in linking.

But if you're not doing something tricky like that, about the only suggestion I can make is to set a breakpoint in the _rw::__null_string_ref_rep destructor and make sure it is only called once. If not, you need to get a stack trace and find where both come from.

In your above example, you would set a breakpoint at:
b * 0xc0000000014dccac - 0x34

[FRESNEDA Christine]

This problem occurs only on the customer specific system.
Everything is working fine on other PA-RISc systems.

Any recommended patch for libc to be installed?

Any system parameter?

regards,
Christine

[Dennis Handly]

>Everything is working fine on other PA-RISC systems. Any recommended patch for libc to be installed? Any system parameter?

I've never seen this failure to be related to a specific machine or patch. It's always consistent. Unless the customer can customize the application by dynamically loading more aC++ shlibs?

But you can try the latest aC++ runtime patch: PHSS_33945
http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=2645dec4c7563110VgnVCM100000275d6e10RCRD#b11.11pa

[FRESNEDA Christine]

I have made additional tests today.

I ran the command on the customer system:
/usr/opt/TeMIPV50H/temip/CRB520/mtnm_am/bin/MTNM_AM.exe -e mtnm_am
=>
/usr/lib/pa20_64/dld.sl: Unable to find library 'libit_atli2_ip_acca0331.5'.
Killed

This is the command being ran by temip_enroll executable.
So I looked for the library.

use 'ldd ' and 'chatr ' to see where the library is expected.
Also search with find, where this library is.

Then you can add the path to LD_LIBRARY_PATH or create a link where it's expected and normally the problem should be fixed.
But here it is a bit more tricky.

On customer system:

ux138155# ldd MTNM_AM.exe|grep libit
libit_poa_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_poa_acca0331.5
libit_art_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art_acca0331.5
libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5
libit_poa_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_poa_acca0331.5
libit_art_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art_acca0331.5
libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5
libit_naming_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_naming_acca0331.5
libit_location_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_location_acca0331.5
/usr/opt/temip/orbix/610/shlib/pa20_64/libit_location_acca0331.=> /usr/opt/temip/orbix/610/shlib/pa20_64/libit_location_acca0331.5
Unable to find library 'libit_atli2_ip_acca0331.5'.


ux138155# ls -l /usr/opt/temip/orbix/610/shlib/pa20_64/libit_atli2_ip_acca0331.5*
-r-xr-xr-x 1 root sys 1535464 Jan 23 2004 /usr/opt/temip/orbix/610/shlib/pa20_64/libit_atli2_ip_acca0331.5
-r-xr-xr-x 1 root sys 18 Jan 23 2004 /usr/opt/temip/orbix/610/shlib/pa20_64/libit_atli2_ip_acca0331.5.dependencies


# more /usr/opt/temip/orbix/610/shlib/pa20_64/libit_atli2_ip_acca0331.5.dependencies
it_art.5 it_ifc.5

# ls -l /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art*
-r-xr-xr-x 1 root sys 21272728 Jan 23 2004 /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art_acca0331.5

# ls -l /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5
lrwxr-xr-x 1 root sys 40 Sep 15 2005 /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 -> ../../posix/pa20_64/libit_ifc_acca0331.5

# ls -l /usr/opt/temip/orbix/610/shlib/posix/pa20_64/libit_ifc*
-r-xr-xr-x 1 root sys 1167752 Jan 23 2004 /usr/opt/temip/orbix/610/shlib/posix/pa20_64/libit_ifc_acca0331.5

ux138155# chatr MTNM_AM.exe
MTNM_AM.exe:
64-bit ELF executable
shared library dynamic path search:
LD_LIBRARY_PATH enabled first
SHLIB_PATH enabled second
embedded path enabled third /usr/opt/tns/lib:/usr/opt/temip/lib:/usr/opt/temip/CRB520/lib:/usr/opt/temip/orbix/610/shlib/pa20_64:/usr/opt/temip/orbix/610/shlib/default/pa20_64
shared library list:
libtemip_crb_ec_base.sl
libAMLibCRBinit.sl
libxerces-c1_7_0.sl
libtemip_crb_sl.sl
libit_poa_acca0331.5
libit_art_acca0331.5
libit_ifc_acca0331.5
libtfc_sp.sl
libtfc_wp.sl
libtfc_fmk.sl
libtfc_event_filter.sl
libtemip_event_filter.sl
libmcc_parpml.sl
libmcc_exec.sl
libmcc_sys.sl
libtls76-m.sl
libtns.sl
libm.2
libpthread.1
libnsl.1
libstd_v2.2
libCsup_v2.2
libcl.2
libc.2
libdl.1
shared library binding:
deferred
global hash table disabled
global hash table size 1103
shared library mapped private disabled
shared library segment merging disabled
shared vtable support disabled
explicit unloading disabled
linkage table protection disabled
segments:
index type address flags size
6 text 4000000000000000 z---c- D (default)
7 data 8000000100000000 ---m-- D (default)
executable from stack: D (default)
static branch prediction disabled
kernel assisted branch prediction enabled
lazy swap allocation for dynamic segments disabled
nulptr references disabled
ux138155#


# swlist|grep PHSS
PHSS_28435 1.0 linker startup code / SLLIC ELF support
PHSS_30970 1.0 ld(1) and linker tools cumulative patch
PHSS_32573 1.0 HP aC++ -AA runtime libraries (aCC A.03.61)


On my test system:

No PHSS for aC++ libraries is installed

hassin,fresneda> ldd MTNM_AM.exe | grep libit_
libit_poa_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_poa_acca0331.5
libit_art_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art_acca0331.5
libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5
libit_poa_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_poa_acca0331.5
libit_art_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_art_acca0331.5
libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5
libit_naming_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_naming_acca0331.5
libit_location_acca0331.5 => /usr/opt/temip/orbix/610/shlib/pa20_64/libit_location_acca0331.5
libit_location_acca0331.5 => /opt/iona/shlib/pa20_64/libit_location_acca0331.5
libit_atli2_ip_acca0331.5 => /opt/iona/shlib/pa20_64/libit_atli2_ip_acca0331.5
libit_atli2_acca0331.5 => /opt/iona/shlib/pa20_64/libit_atli2_acca0331.5
libit_key_replacer_stubs_acca0331.5 => /opt/iona/shlib/pa20_64/libit_key_replacer_stubs_acca0331.5
libit_art_acca0331.5 => /opt/iona/shlib/pa20_64/libit_art_acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5
/usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_acca0331.5 => /usr/opt/temip/orbix/610/shlib/default/pa20_64/libit_ifc_
acca0331.5

hassin,fresneda> ls -l /opt/iona/shlib/pa20_64/libit_atli2_ip_acca0331.5
-r-xr-xr-x 1 root sys 1624248 Aug 17 2007 /opt/iona/shlib/pa20_64/libit_atli2_ip_acca0331.5

/usr/opt/TeMIPV50H/temip/orbix/6X/shlib/pa20_64
=>
hassin,fresneda> ls -l libit_atli2_ip_acca0331.5*
-r-xr-xr-x 1 root sys 1624248 Aug 17 2007 libit_atli2_ip_acca0331.5
-r-xr-xr-x 1 root sys 18 Aug 17 2007 libit_atli2_ip_acca0331.5.dependencies


hassin,fresneda> chatr MTNM_AM.exe | more
MTNM_AM.exe:
64-bit ELF executable
shared library dynamic path search:
LD_LIBRARY_PATH enabled first
SHLIB_PATH enabled second
embedded path enabled third /usr/opt/tns/lib:/usr/opt/temip/lib:/usr/opt/temip/CRB520/lib:/usr/opt/temip/orbix/610/shlib/pa20_
64:/usr/opt/temip/orbix/610/shlib/default/pa20_64
shared library list:
libtemip_crb_ec_base.sl
libAMLibCRBinit.sl
libxerces-c1_7_0.sl
libtemip_crb_sl.sl
libit_poa_acca0331.5
libit_art_acca0331.5
libit_ifc_acca0331.5
libtfc_sp.sl
libtfc_wp.sl
libtfc_fmk.sl
libtfc_event_filter.sl
libtemip_event_filter.sl
libmcc_parpml.sl
libmcc_exec.sl
libmcc_sys.sl
libtls76-m.sl
libtns.sl
libm.2
libpthread.1
libnsl.1
libstd_v2.2
libCsup_v2.2
libcl.2
libc.2
libdl.1
shared library binding
deferred
global hash table disabled
global hash table size 1103
shared library mapped private disabled
shared library segment merging disabled
shared vtable support disabled
explicit unloading disabled
linkage table protection disabled
segments:
index type address flags size
6 text 4000000000000000 z---c- D (default)
7 data 8000000100000000 ---m-- D (default)
executable from stack: D (default)
static branch prediction disabled
kernel assisted branch prediction enabled
lazy swap allocation for dynamic segments disabled
nulptr references disabled

[Dennis Handly]

/usr/lib/pa20_64/dld.sl: Unable to find library 'libit_atli2_ip_acca0331.5'.
Killed

Is this a new problem?

>But here it is a bit more tricky.

What was the tricky part?

>No PHSS for aC++ libraries is installed

Then you should probably install the latest.

[FRESNEDA Christine]

hello,

No it is not a new problem.
But I have defined LD_LIBRARY_PATH so that now the 'ldd -s' finds correctly all shared libraries.
despite this I still get a SEGV when the process is finishing.

I double checked my test system where I have no problem at all and here is what is installed:
# Bundle(s):
#

B3901BA B.11.11.16 HP C/ANSI C Developer's Bundle for HP-UX (S800)
B3913DB C.03.70 HP aC++ Compiler (S800)
B6835AA B.11.00.05 Netscape 6.2.1
B6848AB 1.1.2 GTK+ Support Libraries for HP-UX 11.00
BUNDLE B.11.23 Patch Bundle
BUNDLE11i B.11.11.0306.1 Required Patch Bundle for HP-UX 11i, June 2003
Base-VXVM B.03.20.1 Base VERITAS Volume Manager 3.2 for HP-UX
CDE-English B.11.11 English CDE Environment
ECSAuxiliary A.03.32 ECS Auxiliary
ECSAuxiliaryJp A.03.32 ECS Auxiliary (Japanese)
ECSAuxiliarySch A.03.32 ECS Auxiliary (Simplified Chinese)
ECSCmgNNMHP11 A.03.32 ECS Config/Mgmt GUI for NNM (including Japanese support)
ECSComposer A.03.32 Correlation Composer
ECSOrchestrator A.03.20 ECS Orchestrator
FDDI-00 B.11.11.02 PCI FDDI;Supptd HW=A3739A/A3739B;SW=J3626AA
FibrChanl-00 B.11.11.09 PCI/HSC FibreChannel;Supptd HW=A6684A,A6685A,A5158A,A6795A
GOLDAPPS11i B.11.11.0612.459 Applications Patches for HP-UX 11i v1, December 2006
GOLDBASE11i B.11.11.0612.459 Base Patches for HP-UX 11i v1, December 2006

I will ask the customer to upgrade but to be honnest I'm a bit pessimistic.

regards,
Christine

[Dennis Handly]

>here is what is installed: # Bundle(s):

This lumps all of the patches into a bundle so we can't see the aC++ runtime patches.
You would probably need to use:
swlist -l patch "PHSS_*" | grep aC

>I will ask the customer to upgrade but to be honest I'm a bit pessimistic.

Well, it does work on your system. :-)
Other than the patch level, what is different on the customer's system? The same application but different data?
Is there a trivial way to get into your application and then exit, so you can see if that path fails?

[wangnijie8634]

Hi, all:

I meeting the same problem on our system, it coredump on last line of our program "return 0",

$ gdb -c core
Core was generated by `SimpappClient'.
(gdb) where
#0  0xc0000000003f3938 in __cxxTerm_body+0x18 () from /usr/lib/pa20_64/libCsup.2
#1  0xc0000000003f39c8 in __cxxTerm+0x20 () from /usr/lib/pa20_64/libCsup.2
#2  0xc0000000001c3428 in exit+0x80 () from /usr/lib/pa20_64/libc.2
#3  0xc00000000000a084 in UT_exit+0x64 () from /usr/lib/pa20_64/dld.sl
#4  0xc00000000000a0a4 in exit+0x14 () from /usr/lib/pa20_64/dld.sl
#5  0xc00000000000a168 in $START$+0xb0 () from /usr/lib/pa20_64/dld.sl
Error accessing memory address 0x0: Invalid argument.

Does the problem is solved? If it does, how did you guys slove it? Thanks.

[wangnijie8634]

Here is my version of acc

$ /usr/sbin/swlist -l patch "PHSS_*" | grep aC
# PHSS_30414                            1.0            aC++ Runtime (IA: A.05.56, PA: A.03.50)
# PHSS_32511                            1.0            HP aC++ Compiler (A.03.63)

[Dennis Handly]

>#0  0xc0000000003f3938 in __cxxTerm_body+0x18 /usr/lib/pa20_64/libCsup.2

This could occur if you call atexit(3) and you have unloaded the shlib that contains the function.

>PHSS_32511       HP aC++ Compiler (A.03.63)

This is 6+ years old.

[wangnijie8634]

Did you mean that, I have unloaded the shlib that contains function __cxxTerm_body, then I call atexit(3) , so it cordump?

[Dennis Handly]

>I have unloaded the shlib that contains function __cxxTerm_body

No, you have unloaded the atexit registered function that __cxxTerm_body was going to call.

Basically shouldn't be using atexit(3) in C++, especially if you have shlibs.