Core Dump with multi-threaded application...

[Sanjay Sutar]

Hi All,
I have a multi-threaded application which gives core dump on only HPUX box but runs fine on other.

the stack trace doesnt give much information;

(gdb) where
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
#1 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#2 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#3 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#4 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#5 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
Cannot access memory at address 0x2e2f4870
(gdb)

Attached is the output of swlist.

$uname -a
HP-UX hpcmtest B.11.11 U 9000/800 6466272
unlimited-user license

As I couldnt get any useful info from the stack trace , I added few more debug (print) statement in my code after each step and found that the code crashes when it calls fexist

fexist(get_path());

----here is def of get_path() in .h file

string get_path () { return NIS_PWDIR_ + SHADOW; }

--- and def of fexist

bool fexist(const string& path)
{
return (!path.empty() && access(path.c_str(), F_OK) == 0);
}

I also started another thread with opens a socket and calls accept() blocking call....

I am not sure which thread is causing crash, as the main thread also prints 5-6 message and crash just after printing statement before it calls fexist...

Any help or pointers to analyze core file will be appreciated.

P.S. This thread has been moevd from HP-UX > General to HP-UX > languages. - Hp Forum Moderator

[Peter Nikitka]

Hi,

a multithreaded application under HP-UX needs to be compiled AND linked with '-mt' option of aCC - check this first!

mfG Peter

[Sanjay Sutar]

Thanks for your reply!!

I am using g++ for compiling and linking.
Looking at the man g++, I found -mthread option...
Is this equivalent to -mt option for aCC.

[Sanjay Sutar]

I am using following option for g++ ,

-g -threads -c $(INCLUDES) -DDEBUG
-DNISPLUS_STUB -D_hpux -Dunix

[Sanjay Sutar]

And these are my linking options

-lpthread -lcrypt -lxnet -lnsl -lsec -lstdc++ -Xlinker +b -Xlinker ../li
b:. -Xlinker +s

[Peter Nikitka]

Hi,

I really would give -mthread a try for the compiler.
What do you use for a linker:
- do you call 'ld' directly?
- do you use /usr/bin/ld or something out of the gcc stuff?
- let you do g++ the work for linking as well?

In the last case I would add '-mthread' to the linking command option as well.

For the other cases you would have do do additional research and tests. Here is link to appropriate documentation of the aCC compiler:

http://docs.hp.com/en/7762/5991-4874/threads.htm

Look for corresponding information in the gcc documentation.

mfG Peter

[Sanjay Sutar]

Some more information on this;

We had similar crash at same location i.e while concating two strings , earlier... That time we found out that the ICU lib fies were not rebuilt with gcc 3.4.2 with which the application was built...
Rebuilding the ICU lib files with gcc 3.4.2 solved our issue.

But now I dont see anything related to strin g in core file...

[Dennis Handly]

What gdb are you using? It may not understand g++ code. Also you neglected to mention what signal you got?

You may have a thread stack overflow.

[Sanjay Sutar]

Its GNU gdb 5.3
and this is the last line from my log
"Received a segmentation violation"

I rebuild my application with -pthread option and even then I am getting the crash at same location and same stack trace in core file;

Here is the analysis of core file....

[root@sh3hp11i bin]$/usr/local/local/bin/gdb HpuxAgent core_hpcmtest_0206.log
Name collision between 9826 9836
Name collision between gsi dtc300s
Name collision between dm1521 dm1520
Name collision between c108 c108
Name collision between ovt100 vt100
Name collision between 300 dtc300s
Name collision between t1061 t3700
GNU gdb 5.3
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "hppa2.0n-hp-hpux11.00"...
Core was generated by `HpuxAgent'.
Program terminated with signal 11, Segmentation fault.

warning: The shared libraries were not privately mapped; setting a
breakpoint in a shared library will not work until you rerun the program.

Reading symbols from ./HpuxAgent...done.
Reading symbols from ../lib/libAdkApi.sl...done.
Reading symbols from ../lib/libErmApi.sl...done.
Reading symbols from ../lib/libicuuc.sl.26...done.
Reading symbols from ../lib/libicudata.sl.26...done.
Reading symbols from /usr/lib/libpthread.1...done.
Reading symbols from /usr/lib/libxnet.2...done.
Reading symbols from /usr/lib/libnsl.1...done.
Reading symbols from /usr/lib/libxti.2...done.
Reading symbols from /usr/lib/libsec.2...done.
Reading symbols from ../lib/libstdc++.sl.6...done.
Reading symbols from /usr/lib/libm.2...done.
Reading symbols from ../lib/libgcc_s.sl...done.
Reading symbols from /usr/lib/libc.2...done.
Name collision between 9826 9836
Name collision between gsi dtc300s
Name collision between dm1521 dm1520
Name collision between c108 c108
Name collision between ovt100 vt100
Name collision between 300 dtc300s
Name collision between t1061 t3700
---Type to continue, or q to quit---
Reading symbols from /usr/lib/libdld.2...done.
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
(gdb) where
#0 0xcd60b9a0 in setsockopt () from ../lib/libstdc++.sl.6
#1 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#2 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#3 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#4 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
#5 0xcd60b990 in setsockopt () from ../lib/libstdc++.sl.6
Cannot access memory at address 0x2e2f4870
(gdb)


Here are the last few lines of my log

DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path
DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path : value of NIS_PWDIR_ :
DBG:07/02/06 18:09:54 Thread:000001 context.h:: shadow_path : value of SHADOW :
BSE:07/02/06 18:09:54 Thread:000001 Received a segmentation violation...


From the log it seems that there are no values for two variables NIS_PWDIR_ and SHADOW , and the code is crashing when I concanate these two string and return them in my get_path function

Can it be a cause of crash...

[Dennis Handly]

Are you recursive? Do you call setsockopt? If not, your gdb is broken. Why don't you use HP's gdb and see if it is any better.

>and the code is crashing when I concanate these two string and return them in my get_path function
>Can it be a cause of crash?

A thread stack overflow will cause random crashes. But you can't figure this out until you get a debugger that will trace the whole stack.

Or you could just call pthread_default_stacksize_np(3) to make it bigger, just to see.

[Sanjay Sutar]

I used HP gdb and here is the analysis of core dump

sh1hp11i[/sanjay/agent/bin]#/usr/local/opt/langtools/bin/gdb HpuxAgent core_hpcmtest_0206.log
HP gdb 5.2 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00
and target hppa1.1-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.2 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `HpuxAgent'.
Program terminated with signal 11, Segmentation fault.
Unknown si_code. Report to HP.
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
182 /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h: No such file or directory.
in /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h
(gdb) where
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
#1 0x197b4 in _ZStplIcSt11char_traitsIcESaIcEESbIT_T0_T1_ERKS6_S8_ ()
at /usr/local/include/c++/3.4.2/bits/basic_string.h:1984
(gdb)

I have observed this type of core file earlier and that time I had to rebuild the ICU lib files as they were built with older gcc.

Now in this case if I execute following script for all by lib and binary files ;

------------------
cd $1
echo " Cur dir `pwd`"
for i in `ls`
do
# echo "$i"
strings -a $i | grep gcc-3.4.2-b > /dev/null && echo "File $i was built with gcc-3.4.2-b!"
strings -a $i | grep gcc-3.3.1 > /dev/null && echo "File $i was built with gcc-3.3.1!"
strings -a $i | grep gcc-3.4.2 > /dev/null && echo "File $i was built with gcc-3.4.2!"
done

------------------

I get libstdc++.sl.6 file having ref to gcc-3.4.2-b

But this works on all other machine and gives core only on one machine...

Any clue???

Thanks for all your assistance....

[Dennis Handly]

>I used HP gdb and here is the analysis of core dump

Much better. Your two frames just had:
#0 string::string(const string&)
#1 std::operator+(const string&, const string&)

Now you need to do bt. Or did it stop after two frames? Without the complete thread stack trace, I can't tell if you have a thread stack overflow. So either you make the call below or provide more info:
(gdb) info reg
(gdb) disas $pc-4*32 $pc+4*8

>Any clue???

Did you call pthread_default_stacksize_np(1000000) to set the default thread size to 1 Mb?

[Sanjay Sutar]

How you figured out what the frame 0 has?
When I saw into basic_sting.h , line 182, it has some thing else.

And bt also gives me same output.

(gdb) bt
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
#1 0x197b4 in _ZStplIcSt11char_traitsIcESaIcEESbIT_T0_T1_ERKS6_S8_ ()
at /usr/local/include/c++/3.4.2/bits/basic_string.h:1984

I have attached the complete analysis with output of gdb command you asked for.

Thanks a lot for your continued assistance.

[Dennis Handly]

>How you figured out what the frame 0 has?
When I saw into basic_string.h , line 182, it has some thing else.

I demangled the name on IPF with aC++'s c++filt then I replaced the long template name by its typedef, so it would be easy to understand. I assume you got something with basic_string<...>, the copy constructor.

This doesn't look like a thread stack overflow:
;;; Line: 257
0xcd60b990 <_ZNSsC1ERKSs+0xc8>: ldw -0x12c(%sp),%r20
0xcd60b994 <_ZNSsC1ERKSs+0xcc>: ldw 0(%r20),%r20 NULL ptr
0xcd60b998 <_ZNSsC1ERKSs+0xd0>: stw %r20,-0x124(%sp)
0xcd60b99c <_ZNSsC1ERKSs+0xd4>: ldo -0xc(%r20),%r26
;;; Line: 182
0xcd60b9a0 <_ZNSsC1ERKSs+0xd8>: ldw 8(%r26),%r20 <<<< abort here

R20 is 0 and R26 is -4. I'm not sure what's at SP-0x12c, I didn't have you go back far enough. Using "disas $pc", will do the whole function. In any case, you might want to link with -z so you abort sooner. (Or use chatr -z HpuxAgent)

(You have some instructions missing, 0xcd60b948 to 0xcd60b950.)

[Sanjay Sutar]

Here is the complete assembly dump.

I dont know how to read it.

[Dennis Handly]

>>I'm not sure what's at SP-0x12c,

Ok, this comes from R25, the string you are copying from. So either NIS_PWDIR_ or SHADOW has not been initialized. That's why you want to link with -z.

Of course the pointer inside the string could be NULL. Do the following:
(gdb) x /2x $sp-0x12c
(gdb) x /4x *(void**)($sp-0x12c)

[Sanjay Sutar]

(gdb) x/2x $sp-0x12c
0x7aff13d4: 0x400014b0 0x7aff0ed0
(gdb) x/4x *(void**)($sp-0x12c)
0x400014b0 <_ZN7Context10NIS_PWDIR_E>: 0x00000000 0x00000000 0x00000000 0xffff0112

[Dennis Handly]

(gdb) x/2x $sp-0x12c
0x7aff13d4: 0x400014b0 0x7aff0ed0

You are copying the string from 0x400014b0, NIS_PWDIR, to a local temp 0x7aff0ed0

(gdb) x/4x *(void**)($sp-0x12c)
0x400014b0 <_ZN7Context10NIS_PWDIR_E>: 0x00000000 0x00000000 0x00000000 0xffff0112

The contents of Context::NIS_PWDIR_ have either been zeroed out or you are accessing it before it has been initialized because of order of static initialization.

You can put a watch point on it by:
(gdb) watch *(int*)0x400014b0

For being "zeroed out", it should start at 0, then be set valid, then reset to 0, what you are looking for, then you have the abort.
If never valid, you have initialization order issues.

[Sanjay Sutar]

Sorry Dennis, I cant put any watch.
The application is not crashing on my machine and I dont have access to the customer environment where its crashing.

I only have the core file for analysis.

[Dennis Handly]

>I cant put any watch.
>The application is not crashing on my machine and I dont have access to the customer environment where its crashing.

Then there is not much you can do unless you want to write tool to dump ALL of your data structures.

>I only have the core file for analysis.

(I hope you were given all of the user's shlibs? See gdb's packcore command.)

You could look at stack traces for the other threads to see if they are fiddling with NIS_PWDIR_. (I assume you are doing locking?)

You could look at all globals that are "close" to NIS_PWDIR_ to see if there is a pattern for that zeroing. For named globals you can use the following:
$ nm -pxv HpuxAgent

You can use the x command to look at raw memory:
(gdb) x /128x 0x400014b0-64*4

If this isn't helpful, record this info for next time.

[Sanjay Sutar]

>>(I hope you were given all of the user's >>shlibs? See gdb's packcore command.)

I am analyzing the core file on my test machine where it works fine. So its for sure has all the shlib files.

>>You could look at stack traces for the other threads to see if they are fiddling with NIS_PWDIR_

forgive my ignorance , but how do I do that?

>>to see if there is a pattern for that $ nm -pxv HpuxAgent

nothing like pattern of zero


>>(gdb) x /128x 0x400014b0-64*4

Attached is the dump of raw memory...

And this application works fine on most of our test machine but gives core on the customer machine. Is there something specific to that machine I should look for?

Beacause when I run same application both NIS_PWDIR_ and SHADOW are assigned default values as;

string Context::SHADOW = "shadow";
string Context::NIS_PWDIR_ = "/etc/";

And this is not conditional code, and should always initialize both variables. So how come they got zeroed on the customer machine.

Is there anything else I should look at?

As I mentioned earlier , when we had similar problem earlier , we had to rebuilt ICU lib files as they were built with older gcc.

But this time all the component of application are built with same gcc 3.4.2

Any pointers/info is much appreciated.

[Dennis Handly]

>>(I hope you were given all of the user's shlibs? See gdb's packcore command.)

>I am analyzing the core file on my test machine where it works fine. So its for sure has all the shlib files.

That's not what I mean. You must have an EXACT copy of every shlib your customer has. Including libc, etc. It is easier for your customer to just give you what they have with packcore, rather than play guessing games. That may be why your stack traces aren't complete.

>>You could look at stack traces for the other threads to see if they are fiddling with NIS_PWDIR_

>forgive my ignorance, but how do I do that?

To dump all threads in gdb:
(gdb) info thread
(gdb) thread apply all bt

Then look at the functions in each trace to see where they all are. See if there is a pattern the next time it happens. See if any function is accessing any variables near in memory to NIS_PWDIR_.

>>to see if there is a pattern for that $ nm -pxv HpuxAgent

This gives the mangled names of variables in memory order.

>nothing like pattern of zero
>Attached is the dump of raw memory...

There are a bunch of variables that are 0. If they are strings, they are also bad:
0x40001404 0x00000000 0x00000000 0x00000000
0x40001410 <_ZN7Context8DEV_PTS_E>: 0x00000000 0x0 0x0 0x0
0x40001420 <_ZN7Context5PROC_E>: 0x00000000 0x0 0x0 0x0
0x40001430 <_ZN7Context9USR_SBIN_E>: 0x00000000 0x0 0x0 0x0
0x40001440 <_ZN7Context8VAR_RUN_E>: 0x00000000 0x0 0x0 0x0
0x40001450 <_ZN7Context9CRON_DENYE>: 0x00000000 0x0 0x0 0x0
0x40001460 <_ZN7Context8CHPWDAGEE>: 0x00000000 0x0 0x0 0x0
0x40001470 <_ZN7Context11SYSLOG_CONFE>: 0x00000000 0x0 0x0 0x0
0x40001480 <_ZN7Context4GREPE>: 0x00000000 0x0 0x0 0x0
0x40001490 <_ZN7Context7USERMODE>: 0x00000000 0x0 0x0 0x0
0x400014a0 <_ZN7Context22NISMAP_NETGROUP_BYUSERE>: 0x00000000 0x0 0x0 0x0

Then your variable and 2 more words of 0:
0x400014b0 <_ZN7Context10NIS_PWDIR_E>: 0x00000000 0x0 0x0 0xffff0112

(If there are other variables in this area, they aren't listed if static or they don't start on the first address of each line. You need to match up with the nm(1) output.)

>Is there something specific to that machine I should look for?

Nothing is obvious, you'll set it when you find it. ;-) (How many CPUs?)

>So how come they got zeroed on the customer machine.

A bad pointer? Something wasn't locking a critical resource like a pointer?

>As I mentioned earlier, when we had similar problem earlier, we had to rebuilt ICU lib files as they were built with older gcc.

A mismatch in layouts is a possible cause.
What does "info shared" show?

>Any pointers/info is much appreciated.

There isn't much I can suggest except to try to build debugging tools/tracing into your product. Or see if you can get the customer to repeat it. It seems you were able to do that from your original message??

As I said, linking with -z may catch the error sooner.

You are asking questions that are getting into real time and money.

[Sanjay Sutar]

Thanks Dennis for your continued assistance.

I had some luck to reproduce the issue on one of my HPUX box.

I started debugging and was constantly watching the static global variables and they were "zero" from the beginning, I mean from start of the program, in the main function.

Later I did same thing on the box where the application works fine and all the static global variables have proper non-zero values.

And all these static variables are initilized with hard coded values at global declaration area (above all function definition)...

[Dennis Handly]

>I started debugging and was constantly watching the static global variables and they were "zero" from the beginning, I mean from start of the program, in the main function.

These variables should be zero at the very start of executation since they are runtime initialized. But sometime before main, they should be initialized to their proper value.

For PA32, aC++ used to initialized them on a call to _main inside of main. I think g++ does it sooner.

[Sanjay Sutar]

Dennis,
I have written a small sample program that just simulate my situation and it also gives the core dump in same way.

Attached is the sample program.

Here are the commands used to build and link;

g++ -g -pthread -c test.cpp

g++ -o TestStaticString test.o -pthread -L /var/home/HPUXAGENT/lib -L/usr/local/lib -lpthread -lcrypt -lxnet -lnsl -lsec -lstdc++ -Xlinker +b -Xlinker ../lib:. -Xlinker +s -z


Here is the ldd output of the sample exe on build machine.

$ ldd TestStaticString
/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libdld.2 => /usr/lib/libdld.2
/usr/lib/libc.2 => /usr/lib/libc.2
/var/home/HPUXAGENT/lib/libgcc_s.sl => /var/home/HPUXAGENT/lib/libgcc_s.sl
/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/local/lib/libstdc++.sl.6 => /usr/local/lib/libstdc++.sl.6
/usr/lib/libc.2 => /usr/lib/libc.2
/scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/gcc/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/lib/libsec.2 => /usr/lib/libsec.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/lib/libnsl.1 => /usr/lib/libnsl.1
/usr/lib/libxti.2 => /usr/lib/libxti.2
/usr/lib/libxnet.2 => /usr/lib/libxnet.2
/usr/lib/libxti.2 => /usr/lib/libxti.2
/usr/lib/libpthread.1 => /usr/lib/libpthread.1


And here is the ldd output of same sample exe on the test machine where it gives core dump.

bash-2.04# ldd TestStaticString
=>
/usr/lib/libc.2 => ../lib/libc.2
/usr/lib/libdld.2 => ../lib/libdld.2
/usr/lib/libc.2 => ../lib/libc.2
/var/home/HPUXAGENT/lib/libgcc_s.sl => /home/sanjay/agent/lib/libgcc_s.sl
/usr/lib/libc.2 => ../lib/libc.2
/usr/lib/libm.2 => ../lib/libm.2
/usr/local/lib/libstdc++.sl.6 => /home/sanjay/agent/lib/libstdc++.sl.6
/usr/lib/libc.2 => ../lib/libc.2
/scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/gcc/libgcc_s.sl => /home/sanjay/agent/lib/libgcc_s.sl
/usr/lib/libm.2 => ../lib/libm.2
/usr/lib/libsec.2 => ../lib/libsec.2
/usr/lib/libm.2 => ../lib/libm.2
/usr/lib/libnsl.1 => ../lib/libnsl.1
/usr/lib/libxti.2 => ../lib/libxti.2
/usr/lib/libxnet.2 => ../lib/libxnet.2
/usr/lib/libxti.2 => ../lib/libxti.2
/usr/lib/libpthread.1 => ../lib/libpthread.1

bash-2.04# ./TestStaticString
Main started
initialized string variable
Calling getfullname
Segmentation fault (core dumped)
bash-2.04#


ANd the dump analysis;

bash-2.04# gdb TestStaticString core
HP gdb 5.2 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00
and target hppa1.1-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.2 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..

warning: core file may not match specified executable file.
Core was generated by `TestStaticStr'.
Program terminated with signal 11, Segmentation fault.
SEGV_UNKNOWN - Unknown Error
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
182 /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h: No such file or directory.
in /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h
(gdb) where
#0 _ZNSsC1ERKSs ()
at /scratch/njs/pkgbuild/3.3.1/hpux-11/gcc-3.4.2-b/hppa2.0w-hp-hpux11.11/libstdc++-v3/include/bits/basic_string.h:182
#1 0x3b84 in _ZStplIcSt11char_traitsIcESaIcEESbIT_T0_T1_ERKS6_S8_ ()
at /usr/local/include/c++/3.4.2/bits/basic_string.h:1984
(gdb) x/4x *(void**)($sp-0x12c)
0x40001078 <_ZN7static14nameE>: 0x00000000 0x00000000 0xffff0106 0x00000100