self-deadlocking non-threaded C program

[Jon McDermott]

We're having a problem that boils down to a non-threaded program that sees pthread routines being called from within libc functions; and those pthread routines allows a self-deadlocking situation on a mutex is the timing is right.

I've attached an example program that can reproduct the problem by running the program and pressing -C repeatedly until the program hangs. Once it hangs, adopting the process with gdb shows a stack trace where an interrupt handler interupts a mutex call and deadlocks on a mutex_lock as follows:

(gdb) bt
#0 0xc01f0b88 in __ksleep () from /usr/lib/libc.2
#1 0xc04dcbb0 in pthread_mutex_lock () from /usr/lib/libpthread.1
#2 0xc0204138 in __thread_mutex_lock () from /usr/lib/libc.2
#3 0xc017ff9c in _sigfillset () from /usr/lib/libc.2
#4 0xc017e178 in _memset () from /usr/lib/libc.2
#5 0xc0183750 in malloc () from /usr/lib/libc.2
#6 0x25b4 in SIGINT_handler () at threadtest.c:14
#7
#8 0xc04dcc58 in pthread_mutex_lock () from /usr/lib/libpthread.1
#9 0xc0204138 in __thread_mutex_lock () from /usr/lib/libc.2
#10 0xc0180d54 in _sigfillset () from /usr/lib/libc.2
#11 0xc0183900 in free () from /usr/lib/libc.2
#12 0x2750 in main (argc=1, argv=0x7f7f04b4) at threadtest.c:58

The production program needs to link with libpthread because Oracle client software needs it even though we are not using threads explicitly.

The sample program is:

#include
#include
#include

struct sigaction prev_SIGTERM;
long interrupt_count;

void SIGINT_handler()
{
char *p;

++interrupt_count;

p = (char *) malloc (256);
if (p == NULL)
{
printf("Unable to allocate memory in interrupt\n");
exit(0);
}

memset(p, ' ', 32);
free(p);
}

void main(int argc, char *argv[])
{
struct sigaction sigact;
char *p;
int i = 0;
long main_count = 0;

/* Install a signal handler to handle the SIGINT signal. */

sigact.sa_handler = SIGINT_handler;
sigemptyset(&sigact.sa_mask);
sigact.sa_flags = 0;

if(sigaction(SIGINT, &sigact, &prev_SIGTERM))
{
printf("Unable to install signal handler\n");
exit(0);
}

/* Now go into an infinite loop that just makes some system calls. */

for (;;)
{
++main_count;

p = (char *) malloc (256);
if (p == NULL)
{
printf("Unable to allocate memory in main\n");
exit(0);
}

memset(p, ' ', 32);
free(p);

if (++i == 100)
{
i = 0;
printf("Completed %d iterations with %d interrupts\n",
main_count, interrupt_count);
}
}

return;


The compile and link is done with:

cc -c -w -g -Aa -DUNIX -D_HPUX_SOURCE threadtest.c
cc -w -g -Aa -DUNIX -D_HPUX_SOURCE -lpthread threadtest.o -o threadtest

Why are routines like free() and malloc() calling pthread routines when there is only a single thread? Given we have to include libpthread, how can avoid this situation? I'm hoping for some compile/link flags that will turn off the calls the pthread routines.

[Emil Velez]

The functions are calling thread libraries since functions like malloc and free are thread safe. This means that the create thread specific mutexes for their own housekeeping in case there were multiple threads in the program.

[Emil Velez]

Now I know why. Singnal handling with threads needs to be done very carefully. Evidently the library you are using may create threads and a signal can be seen by all threads and it is undefined (by the thread library not HP) which thread sees and acts on a particular signal. The book "threadtime" has a good chapter on signals and threads and I suggest that you look at it. Since you are linking with a thread based oracle library this will continue to be a issue.

Good luck and I hope I am providing a partial answer.

[Jon McDermott]

The frustrating thing is that we are not doing any threaded programming, it's just a single thread. The thread stuff comes into play when we link in libpthread, which is required by Oracle when we link with their client tools.

I was hoping for some specific "Here's a workaround for you" information, like a compile or link flag that turns it all off.

[Geoff_21]

Just incase there are any other lost souls...

We also appeared to hit this problem with mutex deadlocking.

Jons program wasn't signal safe (This is a URL that gives details of signal-safe functions;

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,463,00.html)

We replaced the malloc stuff with simple loops and still had the problem on multi processor systems running HPUX 11.

The answer was in our case was a libc cumulative patch PHCO 28425, (or rather PHCO_20555 that was contained within it)

Hope this helps.

Geoff