Program terminated with signal 11, Segmentation fault. SEGV_MAPERR - Address not mapped to objec

[Make software boundle]

Hi,

I have difficulty to get "pthread_create" to work from another thread.

When I execute the code; the output looks like this:

aps39-63-root# ./mysqlbackup -s test -v -X
...
2011-03-25.23:06:57                                 main::begin backup
2011-03-25.23:06:57            test@mysql           main:pthread_create::<nr=0,thread=1129534518,handle=0>
2011-03-25.23:06:57            test@mysql           dbwriter:start::<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           dbwriter:system:mysqldump --user=*** --single-transaction --routines --flush-logs  test > /tmp/mysql.test:<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           main:pthread_create::<nr=0,thread=2,handle=0>
2011-03-25.23:06:57            test@mysql           tsmwriter:start::<nr=1,thread=3,handle=0>
...
rc=to_tsm(in,p->handle,filespace,dir,file,application,username,dbrc,p->adsm_arch_or_back
2011-03-25.23:07:00            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
Memory fault(coredump)
aps39-63-root#

And If I debuging the corefile using gdb:

aps39-63-root# gdb -core core
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

Reading symbols from mysqlbackup...done.
Core was generated by `mysqlbackup'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object

warning: Load module /opt/tivoli/tsm/client/icc64/icc/icclib/libicclib.so has been stripped.
Debugging information is not available.


warning: Load module /opt/tivoli/tsm/client/icc64/icc/osslib/libcrypto.so.0.9.7 has been stripped.
Debugging information is not available.

#0  0x4000000000016130:0 in to_tsm () at ../to_tsm.c:37

warning: Source file is more recent than executable mysqlbackup.

37      {
(gdb) quit
aps39-63-root#

The strange thing is that if I comment out the line where "pthread_create" is located,

which is later in the code to_tsm() function, the program works without core dump, and will print all the lines before the "pthread_create" command.!?

Here is an extract of the to_tsm() function

#include <ctype.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <limits.h>
#include <time.h>

#include <pthread.h>
#include <unistd.h>
#include <libgen.h>
...

int16 to_tsm(int fd,dsUint32_t handle,char *filespace,char *dir,char *file,char *adsm_info,char *username,int *dbrc,int arch_or_back,dataelement *p)

{

        (some variable declarations)

        pthread_t       threads;
        pthread_attr_t  attr;

                  printf("this works fine\n);

                  sleep(1);

        pthread_attr_init(&attr);

                  printf("this works fine\n);

                  sleep(1);

        #if defined(_OPEN_THREADS)
                detached = PTHREAD_CREATE_JOINABLE
                pthread_attr_setdetachstate(&attr, &detached);
        #endif

                  printf("this works fine\n);

                  sleep(1);

        pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);

                  printf("this works fine\n);

                  sleep(1);

       (some other stuffs, which is not relevant to show here)

                  printf("this works fine\n);

                  sleep(1);

        threadrc=pthread_create( &threads, &attr, readinback, (void*)&r);

        if(threadrc)
        {
               sprintf(msg,"failed, rc=%d",threadrc);
               output(p,"to_tsm","pthread_create",msg,stderr);
        }
 

        (some other codes)

        pthread_join( threads, (void **) &read_offset);

        ...

}

If I comment out the line in "red" text, the code will print all the lines before the "pthread_create", actually the complete code is working.

This code is working fine also on all other plattforms (SUN64 on x86/SPARC, AIX64, Linux 32/64).

Please help me to detect what is actually causing this failure..

The output from the code when the pthread_create is out-commented.

aps39-63-root# ./mysqlbackup -s test -v -X
...

2011-03-25.23:38:17                                 main::begin backup
2011-03-25.23:38:17            test@mysql           main:pthread_create::<nr=0,thread=1129534518,handle=0>
2011-03-25.23:38:17            test@mysql           main:pthread_create::<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           tsmwriter:start::<nr=1,thread=3,handle=0>
...

2011-03-25.23:38:17            test@mysql           dbwriter:start::<nr=0,thread=2,handle=0>
...

2011-03-25.23:38:17            test@mysql           tsmwriter:to_tsm::<nr=1,thread=3,handle=1>
2011-03-25.23:38:17            test@mysql           dbwriter:end:rc=0:<nr=0,thread=2,handle=0>
2011-03-25.23:38:17            test@mysql           dbwriter:pthread_exit::<nr=0,thread=2,handle=0>
2011-03-25.23:38:18            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:19            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
2011-03-25.23:38:20            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
rc=to_tsm(in,p->handle,filespace,dir,file,application,username,dbrc,p->adsm_arch_or_back
to_tsm(7,1,/mysql/test,/,/full,MySQL,root,0,66)
----
............
2011-03-25.23:38:21            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:24            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:25            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
............
2011-03-25.23:38:27            test@mysql           main:while(running(1)>=parallel(1)):sleep(1):<nr=0,thread=2,handle=0>
this is working

...

2011-03-25.23:38:31            test@mysql           tsmwriter:end::<nr=1,thread=3,handle=1>
...

2011-03-25.23:38:31            test@mysql           tsmwriter:pthread_kill::<nr=1,thread=3,handle=1>
2011-03-25.23:38:31            test@mysql           tsmwriter:pthread_exit::<nr=1,thread=3,handle=1>
2011-03-25.23:38:32                                 main:pthread_attr_destroy:
2011-03-25.23:38:32                                 main:pthread_join:
2011-03-25.23:38:32                                 main:pthread_join:
...

aps39-63-root#

Conclusion, I have difficulty to start a thread from a thread.

Please help me.

Regards Tomas

[Dennis Handly]

>I have difficulty to get pthread_create to work from another thread.

Do you have a thread stack overflow?

>#0  0x4000000000016130:0 in to_tsm to_tsm.c:37

A stacktrace would be helpful.  (bt)

If line 37 is the start of the function, you could also have a RSE stackoverflow.

>and will print all the lines before the pthread_create command?

To make sure your buffers are flushed before you abort, you should use stderr:

fprintf(stderr, "this works fine\n);

[Make software boundle]

Thanks for quick reply Dennis,

Do you have a thread stackoverflow?

- How do I know that I have a stackoverflow; as this is what I suspects?

A stacktrace would be helpful.  (bt)

- How do I enable "stacktrace" ?

If line 37 is the start of the function, you could also have a RSE stackoverflow.

- Yes line 37 is the start of the function.

- But how do I find a RSE stackoverflow?

To make sure your buffers are flushed before you abort, you should use stderr:

fprintf(stderr, "this works fine\n);

- I will try to put this in the code, but I have sleep(1) in the code, and the code is coredumping immediatly without the sleeps...

Looking forward for a reply

Regards Tomas

[Dennis Handly]

>How do I know that I have a stackoverflow; as this is what I suspect?

You will be aborting on a normally impossible condition and instruction.

>How do I enable "stacktrace"?

I mentioned it, use the bt command.

Use:

(gdb) bt

(gdb) disas $pc-16*8 $pc+16*4

(gdb) info reg

Once you get a stacktrace, you can compute the differences between the values of $sp on the first and last frames.  The same with $bsp.

>how do I find a RSE stackoverflow?

You would abort on the alloc instruction.  And $bsp would be bad.

>I have sleep(1) in the code, and the code is coredumping immediately without the sleeps

Having the fprintf would be easier to determine where you are than waiting for pauses.

[Make software boundle]

Thanks for explanation.

But I am (unfortunally) not used to the gdb tool.

Could you give me some hints?

Here is the output from the commands you mentioned.

See attachment: gdb.txt

;;; File: ../to_tsm.c
;;;  38 {
0x40000000000164b0:0 <to_tsm+0x10>:
          mov              ret2=0x40c00                                      MMI,
0x40000000000164b0:1 <to_tsm+0x11>:
          addl             r18=0xfffffffffffffed8,gp
0x40000000000164b0:2 <to_tsm+0x12>:           mov              r40=gp;;
0x40000000000164c0:0 <to_tsm+0x20>:
          ld8.acq          r29=[r18]                                         MMI
0x40000000000164c0:1 <to_tsm+0x21>:           sub              sp=sp,ret2
0x40000000000164c0:2 <to_tsm+0x22>:           mov              r21=b2
0x40000000000164d0:0 <to_tsm+0x30>:
          adds             ret3=-80,ret1                                     MMI,
0x40000000000164d0:1 <to_tsm+0x31>:           adds             r15=-72,ret1
0x40000000000164d0:2 <to_tsm+0x32>:           adds             r41=-48,ret1;;
0x40000000000164e0:0 <to_tsm+0x40>:
          st8              [ret3]=r20,16                                     MMI---Type <return> to continue, or q <return> to quit---

0x40000000000164e0:1 <to_tsm+0x41>:           adds             r25=68,r41
0x40000000000164e0:2 <to_tsm+0x42>:           mov              r24=b3
0x40000000000164f0:0 <to_tsm+0x50>:
          adds             r19=32,sp                                         MMI,
0x40000000000164f0:1 <to_tsm+0x51>:           addl             r31=0x168,gp
0x40000000000164f0:2 <to_tsm+0x52>:           adds             r26=72,r41;;
0x4000000000016500:0 <to_tsm+0x60>:
          st8              [r15]=r21,16                                      MMI
0x4000000000016500:1 <to_tsm+0x61>:           adds             r17=8,r18
0x4000000000016500:2 <to_tsm+0x62>:           mov              r28=b4
0x4000000000016510:0 <to_tsm+0x70>:
          addl             r30=0x260,gp                                      MMI,
0x4000000000016510:1 <to_tsm+0x71>:           mov              r27=-184
0x4000000000016510:2 <to_tsm+0x72>:           mov              r23=-232;;
0x4000000000016520:0 <to_tsm+0x80>:
          ld4              r54=[r25]                                         MMI
0x4000000000016520:1 <to_tsm+0x81>:           ld8              r51=[r26]
0x4000000000016520:2 <to_tsm+0x82>:           mov              r42=rp
---Type <return> to continue, or q <return> to quit---
0x4000000000016530:0 <to_tsm+0x90>:
          st8              [ret3]=r24,0                                      MMI,
0x4000000000016530:1 <to_tsm+0x91>:           stfd             [r19]=f0
0x4000000000016530:2 <to_tsm+0x92>:
          mov              r16=0xfffffffffffffd48;;
0x4000000000016540:0 <to_tsm+0xa0>:
          ld8              r50=[r31]                                         MMI
0x4000000000016540:1 <to_tsm+0xa1>:           ld8              r49=[r30]
0x4000000000016540:2 <to_tsm+0xa2>:           mov              r43=pr
0x4000000000016550:0 <to_tsm+0xb0>:
          st8              [r15]=r28,0                                       MMI,
0x4000000000016550:1 <to_tsm+0xb1>:
          addl             r44=0xfffffffffffffe28,gp
0x4000000000016550:2 <to_tsm+0xb2>:           mov              r14=r40;;
0x4000000000016560:0 <to_tsm+0xc0>:
          ld8              gp=[r17]                                          MMI
0x4000000000016560:1 <to_tsm+0xc1>:           add              r41=r27,r40
0x4000000000016560:2 <to_tsm+0xc2>:           mov              b7=r29
End of assembler dump.

(gdb) info reg <-- see attachment

[Make software boundle]

(gdb) info reg
pr0: 0x1
pr1: 0
pr2: 0
pr3: 0
pr4: 0
pr5: 0
pr6: 0
pr7: 0x1
pr8: 0
pr9: 0x1
pr10: 0x1
pr11: 0x1
pr12: 0
pr13: 0
pr14: 0
pr15: 0x1
pr16: 0
pr17: 0
pr18: 0
pr19: 0
pr20: 0
pr21: 0
pr22: 0
---Type <return> to continue, or q <return> to quit---
pr23: 0
pr24: 0
pr25: 0
pr26: 0
pr27: 0
pr28: 0
pr29: 0
pr30: 0
pr31: 0
pr32: 0
pr33: 0
pr34: 0
pr35: 0
pr36: 0
pr37: 0
pr38: 0
pr39: 0
pr40: 0
pr41: 0
pr42: 0
pr43: 0
pr44: 0
pr45: 0
---Type <return> to continue, or q <return> to quit---
pr46: 0
pr47: 0
pr48: 0
pr49: 0
pr50: 0
pr51: 0
pr52: 0
pr53: 0
pr54: 0
pr55: 0
pr56: 0
pr57: 0
pr58: 0
pr59: 0
pr60: 0
pr61: 0
pr62: 0
pr63: 0
gr0: 0
gr1: 0x6000000000000438
gr2: 0x87ffffff7f767c00
gr3: 0x87ffffff7f767c00
gr4: 0
---Type <return> to continue, or q <return> to quit---
gr5: 0
gr6: 0
gr7: 0
gr8: 0x30
gr9: 0x87ffffff7f7bc9c0
gr10: 0x40c00
gr11: 0x87ffffff7f7bc980
gr12: 0x87ffffff7f77bdc0
gr13: 0x60000000000a3200
gr14: 0xe000000142435300
gr15: 0x87ffffff7f7bc988
gr16: 0x6000000000026957
gr17: 0x6000000000000318
gr18: 0x6000000000000310
gr19: 0x87ffffff7f77bde0
gr20: 0xc000000000360b60
gr21: 0
gr22: 0x87ffffff7f768880
gr23: 0xffffffffffffff18
gr24: 0x4000000000025af0
gr25: 0x87ffffff7f7bc9d4
gr26: 0x87ffffff7f7bc9d8
gr27: 0xffffffffffffff48
---Type <return> to continue, or q <return> to quit---
gr28: 0xc000000000354e60
gr29: 0xc000000001fc5b60
gr30: 0x6000000000000698
gr31: 0x60000000000005a0
gr32: 0x7
gr33: 0x1
gr34: 0x87ffffff7f7bcbc0
gr35: 0xc000000000000d22
gr36: 0x87ffffff7f7bcab0
gr37: 0x4000000000004cb0
gr38: 0x87fffffffffffe75
gr39: 0x600000000004e904
gr40: 0x6000000000000438
gr41: 0x87ffffff7f7bc990
gr42: 0x40000000000191e0
gr43: 0x87ffffff7f7bc950
gr44: 0x87ffffff7f7bc990
gr45: 0x87ffffffef746af0
gr46: 0x87ffffffef748590
gr47: 0x30
gr48: 0x6000000000026908
gr49: 0x87ffffffef742278
gr50: 0x87ffffffef746b30
---Type <return> to continue, or q <return> to quit---
gr51: 0x600000000004e940
gr52: 0xc000000000000996
gr53: 0xc00000000035cc10
gr54: 0x42
gr55: 0x87ffffffef7575e8
gr56: 0xc000000000000205
gr57: 0xc000000000388f70
gr58: 0x8e43
gr59: 0x87ffffff7f7bc920
gr60: 0x87ffffffef7575e0
gr61: 0x87ffffffef7572d8
gr62: 0x87ffffffef7572d0
gr63: 0x87ffffffef7570e8
gr64: 0x18
gr65: 0x6000000000084a6d
gr66: 0x6000000000084a9c
gr67: 0x87ffffff7f7bc924
gr68: 0x6000000000084a88
gr69: 0x87ffffffef757820
gr70: 0x87ffffffef756880
br0: 0x40000000000191e0
br1: 0xc000000000360b60
br2: 0
---Type <return> to continue, or q <return> to quit---
br3: 0x4000000000025af0
br4: 0xc000000000354e60
br5: 0xc000000000297ce0
br6: 0xc000000000168780
br7: 0xe00000010b4020c0
rsc: 0x1f
bsp: 0x87ffffff7f77c148
bspst: 0x87ffffff7f77c078
rnat: 0
ccv: 0
unat: 0
fpsr: 0x9804c8a70433f
pfs: 0xc000000000000d22
(sor:0, sol:26, sof:34)
lc: 0
ec: 0
ip: 0x4000000000016530:1
cfm: 0x10a7
(sor:0, sol:33, sof:39)
psr: 0
(gdb)

[Dennis Handly]

>But I am (unfortunately) not used to the gdb tool.

(This is not a typical use of gdb.  This is assembly language debugging.)

Basically your frame is too large and you have a thread stack overflow:

0x40000000000164b0:0 <to_tsm+0x10>:    mov    ret2=0x40c00

0x40000000000164c0:1 <to_tsm+0x21>:    sub    sp=sp,ret2

0x40000000000164f0:0 <to_tsm+0x50>:    adds   r19=32,sp

0x4000000000016530:1 <to_tsm+0x91>:    stfd   [r19]=f0

Your framesize is: 0x40c00 == 265216

This is already twice the default thread stacksize.

[Make software boundle]

Many thanks for quick reply.

So how do I increase the stack size on HP-UX?

Do I have to do this from SAM -> kernel configurations -> maxsize ?

Or is there other ways?

maxssiz                     Dynamic     8388608     8388608     8388608     9.4%      vm
maxssiz_64bit               Dynamic     268435456   268435456   268435456   0.0%      vm

Is there an equivalent file to control the limits as in /etc/limits.conf or /etc/security/limits.conf as in other linux/unix dialects?

My current stack settings from "ulimit -a" shows:

aps39-63-root# ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         1048576
stack(kbytes)        8192
memory(kbytes)       unlimited
coredump(blocks)     4194303
nofiles(descriptors) 512
aps39-63-root#

aps39-63-root# kcusage |grep maxdsi
maxdsiz              11403264 / 1073741824
maxdsiz_64bit        57016320 / 4294967296

aps39-63-root# kcusage |grep maxssiz
maxssiz                786432 / 8388608
maxssiz_64bit           90112 / 268435456
aps39-63-root#

[Make software boundle]

I tried to increase both maxsiz (see above) to the double and the ulimit -s to 3x.
But the code still core dumps.
I am not sure if I did the right action.

Regards Tomas

[Dennis Handly]

>So how do I increase the stack size on HP-UX?

You need to change your application to call pthread_attr_setstacksize.  This isn't the main thread, controlled by kernel parm maxssiz.

>Is there an equivalent file to control the limits as in /etc/limits.conf or /etc/security/limits.conf as in other Linux/unix dialects?

I don't see "thread" for Linux.

>My current stack settings from "ulimit -a" shows:

This is fine, don't change it.

Similar aborts:

http://h30499.www3.hp.com/t5/Languages-and-Scripting/thread-invokes-library-function-obtains-quot-SEGV-MAPERR-Address/m-p/5166770/

http://h30499.www3.hp.com/t5/Languages-and-Scripting/Core-dump-with-SEGV-MAPERR-Address-not-mapped-to-object/m-p/4696656/

The latter shows exporting PTHREAD_DEFAULT_STACK_SIZE will help.