Languages and Scripting

Attempt to free unallocated or already freed object inside fork() in a multithreaded program

 
SOLVED
Go to solution
blackwater
Regular Advisor

Attempt to free unallocated or already freed object inside fork() in a multithreaded program

My multithreaded program in one of its threads calls fork() and then does execle(). When I use libc malloc and free everything iis OK. When I use libRTC in order to check possible invalid memory allocations and deallocations I see this report:

 

>more has_serverd.1748.mem
-----------------------------------------
Attempt to free unallocated or already freed object at 0x9fffffffbdca13b0

Current context is:
(0)  0xc00000000bbae760  print_stack_trace_to_log_file + 0x2c0 at ../../../Src/gnu/gdb/infrtc.c:1557 [/opt/langtools/lib/hpux64/librtc.so]
(1)  0xc00000000bbb2640  __rtc_event + 0x1c0 at ../../../Src/gnu/gdb/infrtc.c:2177 [/opt/langtools/lib/hpux64/librtc.so]
(2)  0xc00000000bbba2b0  rtc_record_free + 0x500 at ../../../Src/gnu/gdb/infrtc.c:3969 [/opt/langtools/lib/hpux64/librtc.so]
(3)  0xc00000000bbbcd20  free + 0x200 at ../../../Src/gnu/gdb/infrtc.c:4484 [/opt/langtools/lib/hpux64/librtc.so]
(4)  0xc000000000203570  __fork_cleanup_channel + 0x90 at /ux/core/libs/threadslibs/src/common/pthreads/sleep.c:3684 [/lib/hpux64/libpthread.so.1]
(5)  0xc000000000157d30  __pthread_fork_child_cleanup + 0x370 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:8018 [/lib/hpux64/libpthread.so.1]
(6)  0xc000000000173010  __pthread_fork + 0x430 at /ux/core/libs/threadslibs/src/common/pthreads/fork.c:1056 [/lib/hpux64/libpthread.so.1]
(7)  0xc00000000055bd90  _fork + 0xd0 [/lib/hpux64/libc.so.1]
(8)  0xc0000000355cbf90  _ZN13Cache_Storage22CouchbaseLoaderStarter5startEv + 0x30 at ucouchbasestorage.cpp:314 [/import/home/sergey.kurenkov/has.051.02.hpux.bc/libexec/cache.so]
(9)  0xc000000035601c30  _ZN8selfcare11CCacheQuery9do_reopenERN6oracle14oracle_connectERKSbIwSt20encoding_char_traitsIwESaIwEERN5boost6tuples5tupleIRKNSA_8hash_mapIS7_S7_NSA_4hashIS7_EEEERKSt8multimapIS7_S7_S
t4lessIS7_ESaISt4pairIS8_S7_EEERSG_RN11db_id_types4baseIxLj22EEERSP_SS_SI_SI_SR_NSB_9null_typeEEERKNS_16operation_args_tEb + 0x1f40 at ucachequery.cpp:923 [/import/home/sergey.kurenkov/has.051.02.hpux.bc/libe
xec/cache.so]
(10) 0xc000000035608350  _ZN8selfcare11CCacheQuery7do_initERN6oracle14oracle_connectERKSbIwSt20encoding_char_traitsIwESaIwEERN5boost6tuples5tupleIRKNSA_8hash_mapIS7_S7_NSA_4hashIS7_EEEERKSt8multimapIS7_S7_St4
lessIS7_ESaISt4pairIS8_S7_EEERSG_RN11db_id_types4baseIxLj22EEERSP_SS_SI_SI_SR_NSB_9null_typeEEERKNS_16operation_args_tEb + 0x330 at ucachequery.cpp:291 [/import/home/sergey.kurenkov/has.051.02.hpux.bc/libexec
/cache.so]
(11) 0xc00000003563b3e0  _ZN5cache12cache_worker10do_executeEv + 0xf30 at cache_threads.cpp:280 [/import/home/sergey.kurenkov/has.051.02.hpux.bc/libexec/cache.so]
(12) 0xc00000003563c9d0  _ZN5cache12cache_worker7executeEv + 0x200 at cache_threads.cpp:359 [/import/home/sergey.kurenkov/has.051.02.hpux.bc/libexec/cache.so]
(13) 0xc000000032a38690  _ZN7threads11thread_procEPv + 0x1c0 [./libhas_common.so]
(14) 0xc00000000013fb20  __pthread_bound_body + 0x190 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4875 [/lib/hpux64/libpthread.so.1]

 

 

I've got a question. The pointer 0x9fffffffbdca13b0 that is passed to free() seems does not belong to heap and is not allocated inside my program. Where does it come from?

 

 

Moreover I know that if I do vfork() instead of fork() there is no coredump. So is it related to pthread_atfork()? However I do not call pthread_atfork() in my code. There are only two calls to it and they come from __libc_startup()

 

 

(gdb) bt
#0  pthread_atfork ()
    at /ux/core/libs/threadslibs/src/common/pthreads/fork.c:234
#1  0x9fffffffb4469310:0 in __thread_atfork ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/threads/wrappers1.c:672
#2  0x9fffffffb4467000:0 in __libc_init ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/threads/libc_init.c:739
#3  0x9fffffffbf2a7080:0 in __hp__init_libc ()
    at /ux/core/libs/threadslibs/src/common/pthreads/tepv.c:86
#4  0x9fffffffbf2bb270:0 in __pthread_startup ()
    at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:1441
#5  0x9fffffffb42a73e0:0 in __libc_startup ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/csu/init_libc.c:153
#6  0x9fffffffbf7705d0:0 in EM_mark_BOS+0x50 () from /usr/lib/hpux64/dld.so

 

(gdb) bt
#0  pthread_atfork ()
    at /ux/core/libs/threadslibs/src/common/pthreads/fork.c:234
#1  0x9fffffffb4469310:0 in __thread_atfork ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/threads/wrappers1.c:672
#2  0x9fffffffbf181b40:0 in libdl_init () at Libdl/libdl.c:76
#3  0x9fffffffb42a7420:0 in __libc_startup ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/csu/init_libc.c:159
#4  0x9fffffffbf7705d0:0 in EM_mark_BOS+0x50 () from /usr/lib/hpux64/dld.so

 

How to fix this invalid free() that libRTC reports inside fork()? Or is possible that it is a problem in libpthread.so?

 

And finally. librtc seems to cause a coredump in another place:

Core was generated by `has_serverd'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object
#0  mark (first=<not available>,
    last=<not available>, leak_detection=<not available>,
    internal_gc=<not available>, location=PROCESS_STACK)
    at ../../../Src/gnu/gdb/infrtc.c:7667
7667    ../../../Src/gnu/gdb/infrtc.c: No such file or directory.
        in ../../../Src/gnu/gdb/infrtc.c
Breakpoint 1 at 0xc000000000171840:2: file /ux/core/libs/threadslibs/src/common/pthreads/fork.c, line 234 from /lib/hpux64/libpthread.so.1.


(gdb) bt
#0  mark (first=<not available>, last=<not available>,
    leak_detection=<not available>, internal_gc=<not available>,
    location=PROCESS_STACK) at ../../../Src/gnu/gdb/infrtc.c:7667
#1  0xc00000000bbcca60:0 in rtc_leaks_or_dangling_info_common (
    leak_detection=true, internal_gc=false)
    at ../../../Src/gnu/gdb/infrtc.c:8085
#2  0xc00000000bbaa8b0:0 in __rtc_leaks_info ()
    at ../../../Src/gnu/gdb/infrtc.c:8182
#3  0xc00000000bbdeb90:0 in produce_memory_report ()
    at ../../../Src/gnu/gdb/infrtc.c:11016
#4  0xc00000000bbc65e0:0 in rtc_leak_analyze ()
    at ../../../Src/gnu/gdb/infrtc.c:9549
#5  0xc0000000005689f0:0 in __thread_once ()
    at ../../../../../core/libs/libc/shared_em_64_perf/../core/threads/wrappers1.c:467
#6  0xc00000000bbc5dd0:0 in _exit (status=0)
    at ../../../Src/gnu/gdb/infrtc.c:6758
#7  0xc0000000355cdba0:0 in Cache_Storage::CouchbaseLoaderStarter::start (
    this=0x9fffffff5e66e4a8) at cache_source/ucouchbasestorage.cpp:374
#8  0xc000000035601c30:0 in selfcare::CCacheQuery::do_reopen (
    this=<not available>, conn=<not available>, aSQL=<not available>,
    tuple_vars=<not available>, args=@0x6000000002e64688, is_strict=false)
    at cache_source/ucachequery.cpp:923
#9  0xc000000035608350:0 in selfcare::CCacheQuery::do_init (
    this=0x600000000987c690, conn=@0x600000000991be00,
    aSQL=@0x6000000002e646b0, tuple_vars=@0x6000000002e64640,
    args=@0x6000000002e64688, is_strict=false)
    at cache_source/ucachequery.cpp:265
#10 0xc00000003563b3e0:0 in cache::cache_worker::do_execute (
    this=0x6000000000222c90) at cache_source/cache_threads.cpp:280
#11 0xc00000003563c9d0:0 in cache::cache_worker::execute (
    this=0x6000000000222c90) at cache_source/cache_threads.cpp:359
warning: No unwind information found.
 Skipping this library /import/home/sergey.kurenkov/has.051.02.hpux.bc/./lib/libxalanMsg.so.

#12 0xc000000032a38690:0 in threads::thread_proc(void*)+0x1c0 ()
   from ./libhas_common.so
#13 0xc00000000013fb20:0 in __pthread_bound_body ()
    at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4875
(gdb)

 

 

11 REPLIES 11
Dennis Handly
Acclaimed Contributor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>The pointer 0x9fffffffbdca13b0 that is passed to free() seems does not belong to heap and is not allocated inside my program. Where does it come from?

 

It looks like a stack address or possibly a mmap region.

You might be able to use the following for more details:

(gdb) info sym 0x9fffffffbdca13b0

 

>So is it related to pthread_atfork()? I do not call pthread_atfork

 

Very possibly.  It doesn't matter if you call it or not.

 

>Or is possible that it is a problem in libpthread.so?

 

It sure looks like it.  Please contact the Response Center and file a bug.

 

>librtc seems to cause a coredump in another place:

 

Do you have a thread stackoverflow?

 

 

 

blackwater
Regular Advisor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>>librtc seems to cause a coredump in another place:

> Do you have a thread stackoverflow?

For this thread:

 

(gdb) f 13
#13 0xc00000000013fb20:0 in __pthread_bound_body ()
    at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4875
4875    /ux/core/libs/threadslibs/src/common/pthreads/pthread.c: No such file or directory.
        in /ux/core/libs/threadslibs/src/common/pthreads/pthread.c
(gdb) p/x $sp
$2 = 0x9fffffff5e670ff0
(gdb) f 0
#0  mark (first=<not available>, last=<not available>,
    leak_detection=<not available>, internal_gc=<not available>,
    location=PROCESS_STACK) at ../../../Src/gnu/gdb/infrtc.c:7667
7667    ../../../Src/gnu/gdb/infrtc.c: No such file or directory.
        in ../../../Src/gnu/gdb/infrtc.c
(gdb) p/x $sp
$3 = 0x9fffffff5e66d7f0
(gdb) p $2-$3
$4 = 14336

 

14336 is not a lot so I guess it is not a stackoverflow.

 

blackwater
Regular Advisor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

> You might be able to use the following for more details:

> (gdb) info sym 0x9fffffffbdca13b0

 

There is core file for this process so I loaded it in gdb:

(gdb) info sym 0x9fffffffbdca13b0
No symbol matches 0x9fffffffbdca13b0.

And judging from "info shared" this address does not belong to any library.

Dennis Handly
Acclaimed Contributor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>(gdb) p/x $sp

 

There are two stacks for each thread.  You also need to compute the $bsp difference.

 

>14336 is not a lot

 

Yes, doesn't seem like a lot.

You need to look at the machine code for frame 0:

(gdb) info reg

(gdb) disas

 

And see what instruction and what register (address) is being used to store/load.

 

>There is core file for this process so I loaded it in gdb:

>And judging from "info shared"

 

Then it is in a stack or mmap region.

 

Try using: elfdump -S -o core

Then see which region has 0x9fffffffbdca13b0.

blackwater
Regular Advisor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

Dennis, thanks for your response.

> There are two stacks for each thread.  You also need to compute the $bsp difference.
> Try using: elfdump -S -o core
Well, I will do these steps later if it is necessary. But first, I have created a test program and it shows these two errors.
I have three HP-UX 11.31 servers and on all of them I can reproduce them regularly but not always.
Could you please check this test on an HP-UX server?
Actually in the test fork() is just called:

     pid_t pid;
     if((pid = fork()) < 0) {
       printf("an error occurred while forking\n");
     }
     else if(pid == 0) {
       if (verbose) {
         printf("being forked, pid: %d, thread: %d \n", getpid(), pthread_self());
       }
       execv("./invalid_program_name",NULL);
       _exit(0);
     }
     else {
       if (verbose) {
         printf("forked, pid: %d, thread: %d \n", getpid(), pthread_self());
       }
     }

This is how I build a test:
  aCC -mt +DD64 -g test_pthread_key_2.cpp -o test_pthread_key_2.acc
 
This is my rtcconfig:
  check_heap=on
  check_leaks=on
  set heap-check on
  set heap-check frame-count 14

 
This is how I run it:
  BATCH_RTC=on LD_PRELOAD=/opt/langtools/lib/hpux64/librtc.so  ./test_pthread_key_2.acc 1 10 2 0
 
And after running the test sometimes I've got no reports about invalid calls to free(), sometimes I've up to five or six reports like this:
>more test_pthread_key_2.acc.3131.mem
-----------------------------------------
Attempt to free unallocated or already freed object at 0x9fffffffbf6993b0

Current context is:
(0)  0xc00000000bbae760  print_stack_trace_to_log_file + 0x2c0 at ../../../Src/gnu/gdb/infrtc.c:1557 [/opt/langtools/lib/hpux64/librtc.so]
(1)  0xc00000000bbb2640  __rtc_event + 0x1c0 at ../../../Src/gnu/gdb/infrtc.c:2177 [/opt/langtools/lib/hpux64/librtc.so]
(2)  0xc00000000bbba2b0  rtc_record_free + 0x500 at ../../../Src/gnu/gdb/infrtc.c:3969 [/opt/langtools/lib/hpux64/librtc.so]
(3)  0xc00000000bbbcd20  free + 0x200 at ../../../Src/gnu/gdb/infrtc.c:4484 [/opt/langtools/lib/hpux64/librtc.so]
(4)  0xc0000000001375e0  __pthread_alloc + 0x2a80 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:3576 [/usr/lib/hpux64/libpthread.so.1]
(5)  0xc0000000001365e0  __pthread_alloc + 0x1a80 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:3473 [/usr/lib/hpux64/libpthread.so.1]
(6)  0xc000000000157d70  __pthread_fork_child_cleanup + 0x3b0 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:8018 [/usr/lib/hpux64/libpthread.so.1]
(7)  0xc000000000173010  __pthread_fork + 0x430 at /ux/core/libs/threadslibs/src/common/pthreads/fork.c:1056 [/usr/lib/hpux64/libpthread.so.1]
(8)  0xc00000000055bd90  _fork + 0xd0 [/usr/lib/hpux64/libc.so.1]
(9)  0x40000000000020b0  _Z10threadfuncPv + 0x390 at test_pthread_key_2.cpp:32 [./test_pthread_key_2.acc]
(10) 0xc00000000013fb20  __pthread_bound_body + 0x190 at /ux/core/libs/threadslibs/src/common/pthreads/pthread.c:4875 [/usr/lib/hpux64/libpthread.so.1]


And I always get core files after the test with backtrace like this:

 

host: srv2-bl870c-02, OS: HP-UX B.11.31>gdb ./test_pthread_key_2.acc core.3131
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `test_pthread_key_2.acc'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object
#0  mark (first=<not available>, last=<not available>,
    leak_detection=<not available>, internal_gc=<not available>,
    location=PROCESS_STACK) at ../../../Src/gnu/gdb/infrtc.c:7667
7667    ../../../Src/gnu/gdb/infrtc.c: No such file or directory.
        in ../../../Src/gnu/gdb/infrtc.c
(gdb) bt
#0  mark (first=<not available>, last=<not available>,
    leak_detection=<not available>, internal_gc=<not available>,
    location=PROCESS_STACK) at ../../../Src/gnu/gdb/infrtc.c:7667
#1  0xc00000000bbcca60:0 in rtc_leaks_or_dangling_info_common (
    leak_detection=true, internal_gc=false)
    at ../../../Src/gnu/gdb/infrtc.c:8085
#2  0xc00000000bbaa8b0:0 in __rtc_leaks_info ()
    at ../../../Src/gnu/gdb/infrtc.c:8182
#3  0xc00000000bbdeb90:0 in produce_memory_report ()
    at ../../../Src/gnu/gdb/infrtc.c:11016
#4  0xc00000000bbc65e0:0 in rtc_leak_analyze ()
    at ../../../Src/gnu/gdb/infrtc.c:9549
#5  0xc0000000005689f0:0 in __thread_once+0xf0 ()
   from /usr/lib/hpux64/libc.so.1
#6  0xc00000000bbc5dd0:0 in _exit (status=0)
    at ../../../Src/gnu/gdb/infrtc.c:6758
#7  0x40000000000022c0:0 in threadfunc (parm=0x0) at test_pthread_key_2.cpp:40
#8  0xc00000000013fb20:0 in __pthread_bound_body+0x190 ()
   from /usr/lib/hpux64/libpthread.so.1
Current language:  auto; currently c


Dennis Handly
Acclaimed Contributor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>sometimes I've got no reports about invalid calls to free(), sometimes I've up to five or six reports like this:

 

Ok, I get something like that.

 

>And I always get core files after the test with backtrace like this:

 

I haven't got any of those.

 

Using tusc, the bad address to free seems to be allocated here in this mmap:

[29949] sigsetreturn(NULL, 0x6211988, 48640) ............. = 0
[29949] getpid() ......................................... = 29949 (29948)
[29949] sysconf(_SC_THREAD_THREADS_MAX) .................. = 3000
[29949] sysconf(_SC_THREAD_KEYS_MAX) ..................... = 256
[29949] mmap(NULL, 18008, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x9fffffffbd694000
[29949] sysconf(_SC_THREAD_STACK_MIN) .................... = 4096
[29949] mpctl(MPC_GETNUMSPUS, 0, 0) ...................... = 4

blackwater
Regular Advisor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>>Or is possible that it is a problem in libpthread.so?

 

>It sure looks like it.  Please contact the Response Center and file a bug.

 

So I have contacted the Response Center. They do not want to fix what seems to be undefined behaviour in their code (a pointer passed in function free()  does not match a pointer earlier returned  by malloc()). Instead I was told to erase mem files with cron. What a silly advice! Why each HP customer who uses libRTC should organize erasing mem-files produced by libRCT in this situation? Why not fix the undefined behavior in HP libpthread.so?

 

This is an email from HP:

 

 

From: Venkatachalam, Subramanian (BCS) 
Sent: 26 февраля 2014 г. 12:16
To: Schigolev, Sergei
Subject: RE: BL870c libpthread.so problem case 4716260826
 
Sergei,
 
As there’s nothing new on the .mem files you provided – given that your abort is worked around I do not intend to keep this case open for more.
 
The number of .mem files can be removed by setting a cron job, looking for this current pattern.
 
Regards,
 
Subbu
HP Technology Services - BCS (HP-UX)
Hewlett-Packard
Working Hours: Mon to Fri 1230 hrs - 2130 hrs IST, GMT+5:30

 

 

Dennis Handly
Acclaimed Contributor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

>They do not want to fix what seems to be undefined behaviour in their code

 

I would suggest that they just file the bug and let R&D look at it.  Perhaps there is something wrong with librtc and not libpthread?

At least if it is filed, it's in the bug database.

 

>Why each HP customer who uses librtc should organize erasing mem-files produced by librtc in this situation?

 

Well that solution only is useful if that's the only error in the file.  ;-)

And those files are piling up.

 

>Why not fix the undefined behavior in HP libpthread.so?

 

Because you have to figure it out?  ;-)

And everything seems to work unless you're also using librtc.

 

Do you have a way to duplicate the problem without using librtc by monitoring the values sent to free in your test case?

I was able to track it down to a mmap region but not what pointer.

But it would be lots easier for R&D to just look at it.

blackwater
Regular Advisor

Re: Attempt to free unallocated or already freed object inside fork() in a multithreaded program

> I would suggest that they just file the bug and let R&D look at it. 

This is a quote from HP letter I have attached:

 

I do not intend to keep this case open for more

 

>Perhaps there is something wrong with librtc and not libpthread?

I doubt. TCMalloc which also handles malloc() and free() reports me the same problem in this situation (invalid pointer passed to free() that had not been allocated earlier on by malloc()).