HP-UX 11.23, gcc4.3.1, __cxa_throw, application coredumps

[blackwater]

So, we use gcc 4.3.1 on HP-UX 11.23/ia64. On exit my application coredumps.
In preceding releases of the application we used libc malloc() and in this release we use mallocNG malloc().


Our code in one of threads roughly looks like this:
int f()
{
if (condition) throw general_fault();
return 0;
}
int g()
{
try {
f()
} catch (general_fault&) {
std::cout << "Exception!";
}
}

And as I said already the application coredumps.

gdb 6.0 gives this backtrace:

psz_msg=0x9fffffff7f71a510 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ossp_rt_registrator.cpp:13
#13 0x4000000000372eb0:0 in os_sp::app_env_log::sigterm_handler (num=15,
r=) at ../../../src/common/rating_shared/app_env_log.cpp:295
#14
#15 0xc00000000237a870:0 in uwx_decode_uinfo () at uwx_uinfo.c:88
#16 0xc0000000023756d0:0 in uwx_get_frame_info () at uwx_step.c:211
#17 0xc00000000237e460:0 in uwx_restore_markers () at uwx_step.c:347
#18 0xc0000000023a0390:0 in _Unwind_RaiseException ()
at ExceptionHandling.C:449
#19 0xc000000009e3d430 in __cxa_throw (obj=,
tinfo=0x600000000002bda8, dest=0x9fffffffbf64edf0)
at /tmp/gcc-4.3.1.tar.gz/gcc-4.3.1/libstdc++-v3/libsupc++/eh_throw.cc:71
#20 0x4000000000335600:0 in ip::raise (code=2138180472, mess=) at ../../../src/common/rating_lib/ip.cpp:73
#21 0x4000000000337130:0 in ip::general_socket::select (this=,
what=5, sec=, msec=)
at ../../../src/common/rating_lib/ip.cpp:243


If I get rid of throwing an exception in f() and use instead a return code like "return -1; /* -1 means error*/ " I have no coredump.



bash-3.00$ gcc -v
Using built-in specs.
Target: ia64-hp-hpux11.23
Configured with: /tmp/gcc-4.3.1.tar.gz/gcc-4.3.1/configure --host=ia64-hp-hpux11.23 --target=ia64-hp-hpux11.23 --build=ia64-hp-hpux11.23 --prefix=/opt/hp-gcc-4.3.1 --with-gnu-as --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-threads=posix --enable-languages=c,c++ --with-gmp=/proj/opensrc/be/ia64-hp-hpux11.23 --with-mpfr=/proj/opensrc/be/ia64-hp-hpux11.23
Thread model: posix
gcc version 4.3.1 (GCC)


Here is log messages of my application:
09:46:16 Received signal. Received signal : signal number 2 si_errno 0 signal code 0

09:46:16 Received signal. Received SIGTERM : signal number 15 si_errno 0 signal code -1

09:46:16 Received signal. Received SIGBUS : signal number 10 si_errno 0 signal code 1 memory addr 9fffffff7f7177e8

Is it my mistake? I would appreciate any advice. By the way we do not get a coredump when our application exits on HP-UX 11.31/ia64.

[Dennis Handly]

Do you have the latest libunwind patch, PHSS_40541?

>Is it my mistake?

What happens when you compile with aC++?

It could be a code generation bug by g++.

[blackwater]

Dennis, thank for your answer.

PHSS_40541 has been installed after your suggestion. However the application still coredumps on HP-UX 11.23/ia64.

Interestingly we ran a few times (with the same set of our libraries what we used on HP-UX 11.23 and of course with system libraries from HP-UX 11.31) the same application that coredumps on HP-UX 11.23 on HP-UX 11.31 and the application on HP-UX 11.31/ia64 exits without any problem and coredump.

We have this dependencies from system libraries:

libclntsh.so.10.1 => /u01/app/oracle/product/10.2.0/db/lib/libclntsh.so.10.1
libpthread.so.1 => /lib/hpux64/libpthread.so.1
libmallocng.so => /lib/hpux64/libmallocng.so
libm.so.1 => /lib/hpux64/libm.so.1
libunwind.so.1 => /lib/hpux64/libunwind.so.1
libc.so.1 => /lib/hpux64/libc.so.1
libnnz10.so => /u01/app/oracle/product/10.2.0/db/lib/libnnz10.so
librt.so.1 => /usr/lib/hpux64/librt.so.1
libnss_dns.so.1 => /usr/lib/hpux64/libnss_dns.so.1
libdl.so.1 => /usr/lib/hpux64/libdl.so.1
libnsl.so.1 => /usr/lib/hpux64/libnsl.so.1
libuca.so.1 => /usr/lib/hpux64/libuca.so.1
libxti.so.1 => /usr/lib/hpux64/libxti.so.1

[blackwater]

Other dependencies:

libstlport.so.5.1
libboost_thread-gcc-mt-p.so libboost_regex-gcc-mt-p.so
libstdc++.so.6
libgcc_s.so.0 =>


By the way we still haven't ported the application to aCC. It might take some time,

[Dennis Handly]

>we do not get a coredump when our application exits on HP-UX 11.31/ia64.

What is the full stack trace? Are you doing that throw on 11.31?

Does it fail without mallocNG?

[blackwater]

Full backtrace:
---------------
Sometimes it looks like this:

#0 0xc00000000029ffb0:0 in kill+0x30 () from /lib/hpux64/libc.so.1
(gdb) bt
#0 0xc00000000029ffb0:0 in kill+0x30 () from /lib/hpux64/libc.so.1
#1 0xc0000000001c3510:0 in raise ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/raise.c:19
#2 0xc000000000260f30:0 in abort ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/abort_em.c:84
#3 0x4000000000372b40:0 in os_sp::app_env_log::sigbus_handler (num=10,
r=0x9fffffff7f714610)
at ../../../src/common/rating_shared/app_env_log.cpp:281
#4
#5 0xc0000000002a20b0:0 in time+0x30 () from /lib/hpux64/libc.so.1
#6 0xc00000000021f3a0:0 in syslog ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/syslog.c:245
#7 0x400000000033fe70:0 in sys_logger::sys_logger_impl::log_message (
this=0x60000000001036a0, type=6,
psz_format=0x9fffffff7f71b910 "Received SIGTERM : signal number 15 si_errno 0 signal code -1", args=0x9fffffff7f71b8e8)
at ../../../src/common/rating_lib/sys_logger.cpp:156
#8 0x4000000000340070:0 in sys_logger::sys_logger::log_message (

this=0x6000000000103690, type=6,
psz_format=0x9fffffff7f71b910 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ../../../src/common/rating_lib/sys_logger.cpp:203
#9 0x4000000000374260:0 in os_sp::syslog_registrator::registr_signal_handler_msg (this=0x60000000000f8040, type=exit_e, numsig=15, ---Type to continue, or q to quit---
psz_msg=0x9fffffff7f71b910 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ../../../src/common/rating_shared/app_env_log.cpp:40
#10 0x4000000000437240:0 in ossp_rt_registrator::registr_signal_handler_msg (
this=0x60000000000f8040, type=exit_e, numsig=15,
psz_msg=0x9fffffff7f71b910 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ossp_rt_registrator.cpp:13
#11 0x4000000000372f00:0 in os_sp::app_env_log::sigterm_handler (num=15,
r=) at ../../../src/common/rating_shared/app_env_log.cpp:295
#12
#13 0x40000000012f87c0 in .stub+0 () at _ios_base.h:202
warning: Attempting to unwind past bad PC 0x40000000012f87c0
#14 0x4000000000335470:0 in ip::raise (code=4, mess=0x0)
at ../../../src/common/rating_lib/ip.cpp:73
#15 0x4000000000337180:0 in ip::general_socket::select (this=,
what=5, sec=, msec=)
at ../../../src/common/rating_lib/ip.cpp:245
#16 0x4000000000432870:0 in rating_server::listener::execute (
this=0x60000000003d5120) at ./connect_listener.cpp:172
#17 0x4000000000340370:0 in threads::thread::thread_proc (
th=0x60000000003d5120) at ../../../src/common/rating_lib/threads.cpp:246
#18 0x40000000003406f0:0 in start_routine_thread (p_arg=0x60000000003d5120)
at ../../../src/common/rating_lib/threads.cpp:18
#19 0xc000000000328f40:0 in __pthread_bound_body ()
at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4594

Sometimes we get:

(gdb) bt
#0 0xc000000018997fb0:0 in kill+0x30 () from ./libc.so.1
#1 0xc0000000188bb510:0 in raise ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/raise.c:19
#2 0xc000000018958f30:0 in abort ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/abort_em.c:84
#3 0x4000000000372b40:0 in os_sp::app_env_log::sigbus_handler(int,__siginfo*,void*)+0xe0 ()
#4
#5 0xc00000001899a0b0:0 in time+0x30 () from ./libc.so.1
#6 0xc0000000189173a0:0 in syslog ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/syslog.c:245
#7 0x400000000033fe70:0 in sys_logger::sys_logger_impl::log_message(unsigned int,char const*,void*)+0x70 ()
#8 0x4000000000340070:0 in sys_logger::sys_logger::log_message(unsigned int,char const*,...)+0x70 ()
#9 0x4000000000374260:0 in os_sp::syslog_registrator::registr_signal_handler_msg(os_sp::evnt_registrator::type_handler_action_e,int,char const*)+0x80 ()
#10 0x4000000000437240:0 in ossp_rt_registrator::registr_signal_handler_msg(os_sp::evnt_registrator::type_handler_action_e,int,char const*)+0x40 ()
#11 0x4000000000372f00:0 in os_sp::app_env_log::sigterm_handler(int,__siginfo*,void*)+0xb0 ()
#12
#13 0x40000000012f87c0 in asn1::asn1_choice::remove_field(stlp_std::basic_string---Type to continue, or q to quit---~
,stlp_std::allocator > const&) ()
warning: Attempting to unwind past bad PC 0x40000000012f87c0
#14 0x4000000000335470:0 in ip::raise(int,char const*)+0x50 ()
#15 0x4000000000337180:0 in ip::general_socket::select(unsigned short,unsigned int,unsigned int) const+0x4e0 ()
#16 0x4000000000432870:0 in rating_server::listener::execute()+0x300 ()
#17 0x4000000000340370:0 in threads::thread::thread_proc(threads::thread*)
+0x1b0 ()
#18 0x40000000003406f0:0 in start_routine_thread+0x30 ()
#19 0xc0000000132fcf40:0 in __pthread_bound_body ()
at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4594



Are you doing that throw on 11.31?
-----------------------------------
Yes. It was probably not clear from my post so I want to clarify this. We use the same executable file.
We built an executable file using gcc 4.3.1 on HP-UX 11.23 and then we run this executable on HP-UX 11.23 and on HP-UX 11.31.
On HP-UX 11.23 this executable sometimes coredumps on exit and sometimes exits without any problem.
On HP-UX 11-31 the same executable (built on HP-UX 11.23) always exits without any problem.

Building without mallocNG
-------------------------
We haven't built this release of the application without mallocNG so far.

[blackwater]

This kind of message is written on console when the application coredumps:

Either

Received SIGBUS : signal number 10 si_errno 0 signal code 0 memory addr 000000000000001f

or

Received SIGBUS : signal number 10 si_errno 0 signal code 0 memory addr 00000000000000c9

[Dennis Handly]

>Sometimes it looks like this:

Are you using sigaltstack(2)?
Do you have a thread stack overflow?

>Sometimes we get:

Neither of these aborts in uwx_decode_uinfo, I would suggest you debug these where you have the source.

You could have blasted the linkage table.

>On HP-UX 11.23 this executable sometimes coredumps on exit

These stack traces don't like like "exit". Was that done in another thread?

[blackwater]

> Neither of these aborts in uwx_decode_uinfo,

It was here:

#0 0xc00000000029ffb0:0 in kill+0x30 () from /lib/hpux64/libc.so.1
(gdb) bt
#0 0xc00000000029ffb0:0 in kill+0x30 () from /lib/hpux64/libc.so.1
#1 0xc0000000001c3510:0 in raise ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/raise.c:19
#2 0xc000000000260f30:0 in abort ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/abort_em.c:84
#3 0x4000000000372af0:0 in os_sp::app_env_log::sigbus_handler (num=10,
r=0x9fffffff7f712610)
at ../../../src/common/rating_shared/app_env_log.cpp:281
#4
#5 0xc000000000279ab0:1 in T_19_73d5_cl___doprnt_main ()
at ../../../../../core/libs/libc/shared_em_64/../core/stdio/doprnt.c:299
#6 0xc0000000002715f0:0 in _doprnt ()
at ../../../../../core/libs/libc/shared_em_64/../core/stdio/doprnt.c:508
#7 0xc000000000294c60:0 in sprintf ()
at ../../../../../core/libs/libc/shared_em_64/../core/stdio/sprintf.c:37
#8 0xc00000000021f380:0 in syslog ()
at ../../../../../core/libs/libc/shared_em_64/../core/gen/syslog.c:243
#9 0x400000000033fe00:0 in sys_logger::sys_logger_impl::log_message (
this=0x60000000001036a0, type=6,
psz_format=0x9fffffff7f71a510 "Received SIGTERM : signal number 15 si_errno 0 signal code -1", args=0x9fffffff7f71a4e8)
at ../../../src/common/rating_lib/sys_logger.cpp:156
#10 0x4000000000340000:0 in sys_logger::sys_logger::log_message (

---Type to continue, or q to quit---
this=0x6000000000103690, type=6,
psz_format=0x9fffffff7f71a510 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ../../../src/common/rating_lib/sys_logger.cpp:203
#11 0x4000000000374210:0 in os_sp::syslog_registrator::registr_signal_handler_msg (this=0x60000000000f8040, type=exit_e, numsig=15,
psz_msg=0x9fffffff7f71a510 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ../../../src/common/rating_shared/app_env_log.cpp:40
#12 0x4000000000437260:0 in ossp_rt_registrator::registr_signal_handler_msg (
this=0x60000000000f8040, type=exit_e, numsig=15,
psz_msg=0x9fffffff7f71a510 "Received SIGTERM : signal number 15 si_errno 0 signal code -1") at ossp_rt_registrator.cpp:13
#13 0x4000000000372eb0:0 in os_sp::app_env_log::sigterm_handler (num=15,
r=) at ../../../src/common/rating_shared/app_env_log.cpp:295
#14
#15 0xc00000000237a870:0 in uwx_decode_uinfo () at uwx_uinfo.c:88
#16 0xc0000000023756d0:0 in uwx_get_frame_info () at uwx_step.c:211
#17 0xc00000000237e460:0 in uwx_restore_markers () at uwx_step.c:347
#18 0xc0000000023a0390:0 in _Unwind_RaiseException ()
at ExceptionHandling.C:449
#19 0xc000000009e3d430 in __cxa_throw (obj=,
tinfo=0x600000000002bda8, dest=0x9fffffffbf64edf0)
at /tmp/gcc-4.3.1.tar.gz/gcc-4.3.1/libstdc++-v3/libsupc++/eh_throw.cc:71
#20 0x4000000000335600:0 in ip::raise (code=2138180472, mess=) ---Type to continue, or q to quit---
at ../../../src/common/rating_lib/ip.cpp:73
#21 0x4000000000337130:0 in ip::general_socket::select (this=,
what=5, sec=, msec=)
at ../../../src/common/rating_lib/ip.cpp:243
#22 0x4000000000432830:0 in rating_server::listener::execute (
this=0x60000000003d3120) at ./connect_listener.cpp:172
#23 0x4000000000340300:0 in threads::thread::thread_proc (
th=0x60000000003d3120) at ../../../src/common/rating_lib/threads.cpp:246
#24 0x4000000000340680:0 in start_routine_thread (p_arg=0x60000000003d3120)
at ../../../src/common/rating_lib/threads.cpp:18
#25 0xc000000000328f40:0 in __pthread_bound_body ()
at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4594

[Dennis Handly]

>It was here:

From your stack traces:
1) You are in the middle of a throw and you get a SIGTERM. This is your issue.

2) You are in a .stub and you get SIGTERM. This is your issue.

3) You are in remove_field and get an unknown signal.

Basically if you are going to kill your process, you should expect to see SIGTERM in your signal handlers, coming from random locations.