topic Re: Core Dump with multi-threaded application... in Operating System - Linux

Core Dump with multi-threaded application...

Sanjay Sutar — Mon, 07 Oct 2013 04:05:58 GMT

Hi All,
I have a multi-threaded application which gives core dump on only HPUX box but runs fine on other.

the stack trace doesnt give much information;

(gdb) where
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
#1 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#2 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#3 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#4 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#5 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
Cannot access memory at address 0x2e2f4870
(gdb)

Attached is the output of swlist.

$uname -a
HP-UX hpcmtest B.11.11 U 9000/800 6466272
unlimited-user license

As I couldnt get any useful info from the stack trace , I added few more debug (print) statement in my code after each step and found that the code crashes when it calls fexist

fexist(get_path());

----here is def of get_path() in .h file

string get_path () { return NIS_PWDIR_ + SHADOW; }

--- and def of fexist

bool fexist(const string& path)
{
return (!path.empty() && access(path.c_str(), F_OK) == 0);
}

I also started another thread with opens a socket and calls accept() blocking call....

I am not sure which thread is causing crash, as the main thread also prints 5-6 message and crash just after printing statement before it calls fexist...

Any help or pointers to analyze core file will be appreciated.

P.S. This thread has been moevd from HP-UX > General to HP-UX > languages. - Hp Forum Moderator

Re: Core Dump with multi-threaded application...

Peter Nikitka — Mon, 05 Feb 2007 08:55:05 GMT

Hi,

a multithreaded application under HP-UX needs to be compiled AND linked with '-mt' option of aCC - check this first!

mfG Peter

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Mon, 05 Feb 2007 09:05:55 GMT

Thanks for your reply!!

I am using g++ for compiling and linking.
Looking at the man g++, I found -mthread option...
Is this equivalent to -mt option for aCC.

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Mon, 05 Feb 2007 09:16:32 GMT

I am using following option for g++ ,

-g -threads -c $(INCLUDES) -DDEBUG
-DNISPLUS_STUB -D_hpux -Dunix

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Mon, 05 Feb 2007 09:24:32 GMT

And these are my linking options

-lpthread -lcrypt -lxnet -lnsl -lsec -lstdc++ -Xlinker +b -Xlinker ../li
b:. -Xlinker +s

Re: Core Dump with multi-threaded application...

Peter Nikitka — Mon, 05 Feb 2007 09:47:34 GMT

Hi,

I really would give -mthread a try for the compiler.
What do you use for a linker:
- do you call 'ld' directly?
- do you use /usr/bin/ld or something out of the gcc stuff?
- let you do g++ the work for linking as well?

In the last case I would add '-mthread' to the linking command option as well.

For the other cases you would have do do additional research and tests. Here is link to appropriate documentation of the aCC compiler:

http://docs.hp.com/en/7762/5991-4874/threads.htm

Look for corresponding information in the gcc documentation.

mfG Peter

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Tue, 06 Feb 2007 02:10:26 GMT

Some more information on this;

We had similar crash at same location i.e while concating two strings , earlier... That time we found out that the ICU lib fies were not rebuilt with gcc 3.4.2 with which the application was built...
Rebuilding the ICU lib files with gcc 3.4.2 solved our issue.

But now I dont see anything related to strin g in core file...

Re: Core Dump with multi-threaded application...

Dennis Handly — Tue, 06 Feb 2007 05:05:09 GMT

What gdb are you using? It may not understand g++ code. Also you neglected to mention what signal you got?

You may have a thread stack overflow.

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Tue, 06 Feb 2007 07:00:41 GMT

Its GNU gdb 5.3
and this is the last line from my log
"Received a segmentation violation"

I rebuild my application with -pthread option and even then I am getting the crash at same location and same stack trace in core file;

Here is the analysis of core file....

[root@sh3hp11i bin]$/usr/local/local/bin/gdb HpuxAgent core_hpcmtest_0206.log
Name collision between 9826 9836
Name collision between gsi dtc300s
Name collision between dm1521 dm1520
Name collision between c108 c108
Name collision between ovt100 vt100
Name collision between 300 dtc300s
Name collision between t1061 t3700
GNU gdb 5.3
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "hppa2.0n-hp-hpux11.00"...
Core was generated by `HpuxAgent'.
Program terminated with signal 11, Segmentation fault.

warning: The shared libraries were not privately mapped; setting a
breakpoint in a shared library will not work until you rerun the program.

Reading symbols from ./HpuxAgent...done.
Reading symbols from ../lib/libAdkApi.sl...done.
Reading symbols from ../lib/libErmApi.sl...done.
Reading symbols from ../lib/libicuuc.sl.26...done.
Reading symbols from ../lib/libicudata.sl.26...done.
Reading symbols from /usr/lib/libpthread.1...done.
Reading symbols from /usr/lib/libxnet.2...done.
Reading symbols from /usr/lib/libnsl.1...done.
Reading symbols from /usr/lib/libxti.2...done.
Reading symbols from /usr/lib/libsec.2...done.
Reading symbols from ../lib/libstdc++.sl.6...done.
Reading symbols from /usr/lib/libm.2...done.
Reading symbols from ../lib/libgcc_s.sl...done.
Reading symbols from /usr/lib/libc.2...done.
Name collision between 9826 9836
Name collision between gsi dtc300s
Name collision between dm1521 dm1520
Name collision between c108 c108
Name collision between ovt100 vt100
Name collision between 300 dtc300s
Name collision between t1061 t3700
---Type to continue, or q to quit---
Reading symbols from /usr/lib/libdld.2...done.
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
(gdb) where
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
#1 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#2 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#3 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#4 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#5 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
Cannot access memory at address 0x2e2f4870
(gdb)

Here are the last few lines of my log

DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path
DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path : value of NIS_PWDIR_ :
DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path : value of SHADOW :
BSE:07/02/06 18:09:54 Thread:000001 Received a segmentation violation...

From the log it seems that there are no values for two variables NIS_PWDIR_ and SHADOW , and the code is crashing when I concanate these two string and return them in my get_path function

Can it be a cause of crash...

Re: Core Dump with multi-threaded application...

Dennis Handly — Tue, 06 Feb 2007 08:19:44 GMT

Are you recursive? Do you call setsockopt? If not, your gdb is broken. Why don't you use HP's gdb and see if it is any better.

>and the code is crashing when I concanate these two string and return them in my get_path function
>Can it be a cause of crash?

A thread stack overflow will cause random crashes. But you can't figure this out until you get a debugger that will trace the whole stack.

Or you could just call pthread_default_stacksize_np(3) to make it bigger, just to see.

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Tue, 06 Feb 2007 10:15:55 GMT

I used HP gdb and here is the analysis of core dump

sh1hp11i[/sanjay/agent/bin]#/usr/local/opt/langtools/bin/gdb HpuxAgent core_hpcmtest_0206.log
HP gdb 5.2 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00
and target hppa1.1-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.2 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `HpuxAgent'.
Program terminated with signal 11, Segmentation fault.
Unknown si_code. Report to HP.
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
182 /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h: No such file or directory.
in /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h
(gdb) where
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
#1 0x197b4 in _ZStplIcSt11char_traitsIcESaIcEESbIT_T0_T1_ERKS6_S8_ ()
at /usr/local/include/c++/3.4.2/bits/basic_string.h:1984
(gdb)

I have observed this type of core file earlier and that time I had to rebuild the ICU lib files as they were built with older gcc.

Now in this case if I execute following script for all by lib and binary files ;

------------------
cd $1
echo " Cur dir `pwd`"
for i in `ls`
do
# echo "$i"
strings -a $i | grep gcc-3.4.2-b > /dev/null && echo "File $i was built with gcc-3.4.2-b!"
strings -a $i | grep gcc-3.3.1 > /dev/null && echo "File $i was built with gcc-3.3.1!"
strings -a $i | grep gcc-3.4.2 > /dev/null && echo "File $i was built with gcc-3.4.2!"
done

------------------

I get libstdc++.sl.6 file having ref to gcc-3.4.2-b

But this works on all other machine and gives core only on one machine...

Any clue???

Thanks for all your assistance....

Re: Core Dump with multi-threaded application...

Dennis Handly — Tue, 06 Feb 2007 22:09:24 GMT

>I used HP gdb and here is the analysis of core dump

Much better. Your two frames just had:
#0 string::string(const string&)
#1 std::operator+(const string&, const string&)

Now you need to do bt. Or did it stop after two frames? Without the complete thread stack trace, I can't tell if you have a thread stack overflow. So either you make the call below or provide more info:
(gdb) info reg
(gdb) disas $pc-4*32 $pc+4*8

>Any clue???

Did you call pthread_default_stacksize_np(1000000) to set the default thread size to 1 Mb?

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Wed, 07 Feb 2007 01:04:50 GMT

How you figured out what the frame 0 has?
When I saw into basic_sting.h , line 182, it has some thing else.

And bt also gives me same output.

(gdb) bt
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
#1 0x197b4 in _ZStplIcSt11char_traitsIcESaIcEESbIT_T0_T1_ERKS6_S8_ ()
at /usr/local/include/c++/3.4.2/bits/basic_string.h:1984

I have attached the complete analysis with output of gdb command you asked for.

Thanks a lot for your continued assistance.

Re: Core Dump with multi-threaded application...

Dennis Handly — Wed, 07 Feb 2007 01:49:16 GMT

>How you figured out what the frame 0 has?
When I saw into basic_string.h , line 182, it has some thing else.

I demangled the name on IPF with aC++'s c++filt then I replaced the long template name by its typedef, so it would be easy to understand. I assume you got something with basic_string<...>, the copy constructor.

This doesn't look like a thread stack overflow:
;;; Line: 257
0xcd60b990 <_ZNSsC1ERKSs+0xc8>: ldw -0x12c(%sp),%r20
0xcd60b994 <_ZNSsC1ERKSs+0xcc>: ldw 0(%r20),%r20 NULL ptr
0xcd60b998 <_ZNSsC1ERKSs+0xd0>: stw %r20,-0x124(%sp)
0xcd60b99c <_ZNSsC1ERKSs+0xd4>: ldo -0xc(%r20),%r26
;;; Line: 182
0xcd60b9a0 <_ZNSsC1ERKSs+0xd8>: ldw 8(%r26),%r20 <<<< abort here

R20 is 0 and R26 is -4. I'm not sure what's at SP-0x12c, I didn't have you go back far enough. Using "disas $pc", will do the whole function. In any case, you might want to link with -z so you abort sooner. (Or use chatr -z HpuxAgent)

(You have some instructions missing, 0xcd60b948 to 0xcd60b950.)

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Wed, 07 Feb 2007 03:48:44 GMT

Here is the complete assembly dump.

I dont know how to read it.

Re: Core Dump with multi-threaded application...

Dennis Handly — Wed, 07 Feb 2007 04:27:14 GMT

>>I'm not sure what's at SP-0x12c,

Ok, this comes from R25, the string you are copying from. So either NIS_PWDIR_ or SHADOW has not been initialized. That's why you want to link with -z.

Of course the pointer inside the string could be NULL. Do the following:
(gdb) x /2x $sp-0x12c
(gdb) x /4x *(void**)($sp-0x12c)

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Wed, 07 Feb 2007 04:47:21 GMT

(gdb) x/2x $sp-0x12c
0x7aff13d4: 0x400014b0 0x7aff0ed0
(gdb) x/4x *(void**)($sp-0x12c)
0x400014b0 <_ZN7Context10NIS_PWDIR_E>: 0x00000000 0x00000000 0x00000000 0xffff0112

Re: Core Dump with multi-threaded application...

Dennis Handly — Wed, 07 Feb 2007 05:58:45 GMT

(gdb) x/2x $sp-0x12c
0x7aff13d4: 0x400014b0 0x7aff0ed0

You are copying the string from 0x400014b0, NIS_PWDIR, to a local temp 0x7aff0ed0

(gdb) x/4x *(void**)($sp-0x12c)
0x400014b0 <_ZN7Context10NIS_PWDIR_E>: 0x00000000 0x00000000 0x00000000 0xffff0112

The contents of Context::NIS_PWDIR_ have either been zeroed out or you are accessing it before it has been initialized because of order of static initialization.

You can put a watch point on it by:
(gdb) watch *(int*)0x400014b0

For being "zeroed out", it should start at 0, then be set valid, then reset to 0, what you are looking for, then you have the abort.
If never valid, you have initialization order issues.

Re: Core Dump with multi-threaded application...

Sanjay Sutar — Wed, 07 Feb 2007 07:09:59 GMT

Sorry Dennis, I cant put any watch.
The application is not crashing on my machine and I dont have access to the customer environment where its crashing.

I only have the core file for analysis.

Re: Core Dump with multi-threaded application...

Dennis Handly — Sat, 05 Oct 2013 03:33:49 GMT

>I cant put any watch.
>The application is not crashing on my machine and I dont have access to the customer environment where its crashing.

Then there is not much you can do unless you want to write tool to dump ALL of your data structures.

>I only have the core file for analysis.

(I hope you were given all of the user's shlibs? See gdb's packcore command.)

You could look at stack traces for the other threads to see if they are fiddling with NIS_PWDIR_. (I assume you are doing locking?)

You could look at all globals that are "close" to NIS_PWDIR_ to see if there is a pattern for that zeroing. For named globals you can use the following:
$ nm -pxv HpuxAgent

You can use the x command to look at raw memory:
(gdb) x /128x 0x400014b0-64*4

If this isn't helpful, record this info for next time.