topic Re: child process hangs on pthread_mutex_lock in Operating System - HP-UX

child process hangs on pthread_mutex_lock

Fedele Giuseppe — Thu, 21 Oct 2010 09:46:20 GMT

Hello,

I have a C multi-thread process that periodically forks.

Sometimes child process hangs and WDB output says:

c32dabd0 in __ksleep+0x10 () from /usr/lib/libc.2
#1 0xc3b002a4 in __sleep_1x1+0x610 () from /usr/lib/libpthread.1
#2 0xc3af8f98 in __mxn_sleep+0xdd8 () from /usr/lib/libpthread.1
#3 0xc3ac501c in + 0x3800 () from /usr/lib/libpthread.1
#4 0xc3ac59b0 in pthread_mutex_lock+0xd0 () from /usr/lib/libpthread.1
#5 0xc3211540 in __thread_mutex_lock+0x70 () from /usr/lib/libc.2
#6 0xc32164f0 in printf+0x118 () from /usr/lib/libc.2
#7 0x156c0 in Print (TrcLevel=4, LineNum=748, Format=0x241a8 "CHILD PROCESS - ExecAction - CloseSock_3(%d)") at CommFunc.c:1015
#8 0x8f50 in ExecAction (action=0x40016064, loc_phase=0x40001820 "-p") at HAReplicationManager.c:748
#9 0x74b0 in main (arc=1, argv=0x64101000) at HAReplicationManager.c:506

The routine is:

void Print (int TrcLevel, int LineNum, const char * Format, ...)
{
char loc_buff [BF];
va_list UnNamedParams;

va_start (UnNamedParams, Format);

if (trc_level >= TrcLevel)
{
sprintf (loc_buff, "%d %05d %s %d - ", mypid, LineNum, print_date(), trc_level);
vsprintf (loc_buff + (strlen (loc_buff)), Format, UnNamedParams);
printf ("%s\n", loc_buff);
fflush (NULL);
}

va_end (UnNamedParams);

return;
}

and line 1015 is:

printf ("%s\n", loc_buff);

Now I have checked all points in which I use pthread_mutex_lock but I cannot find any relation with the above line.

A further info: "mypid" and "trc_level" are variables defined in father process.

Finally, fork env managed through:
pthread_atfork_err = pthread_atfork(prepare, parent, child);

In which direction can I proceed to investigate?

Many thanks

Giuseppe

Re: child process hangs on pthread_mutex_lock

James R. Ferguson — Thu, 21 Oct 2010 11:59:06 GMT

Hi Giuseppe:

In the manpages for 'thread_safety(5)' it is noted that "Cancellation Points can occur when a thread is executing in the following interfaces...vprintf()..."

Regards!

...JRF...

Re: child process hangs on pthread_mutex_lock

Dennis Handly — Fri, 22 Oct 2010 03:55:02 GMT

>I have a C multi-thread process that periodically forks.

In general, you don't want to do this. What do you do after the fork, exec?

To do this properly, this requires you do create atfork handlers for every resource you lock. So when you fork, these get unlocked in the child. Otherwise the child will think a non-existent thread still has the mutex/resource locked.

>The routine is:
void Print(int TrcLevel, int LineNum, const char * Format, ...)

Your problem is with the libc mutex.
Do you have any other threads active in the child?
If not, this seems like a libc bug, what HP-UX version are you using? Do you have the latest libc patches?

>fork env managed through:
pthread_atfork_err = pthread_atfork(prepare, parent, child);

Ok, it seems you know about pthread_atfork.

Re: child process hangs on pthread_mutex_lock

Fedele Giuseppe — Fri, 22 Oct 2010 06:39:24 GMT

Hi the OS is HP-UX B.11.31 on a itanium server.

Giuseppe

Re: child process hangs on pthread_mutex_lock

Dennis Handly — Fri, 22 Oct 2010 08:30:31 GMT

>the OS is HP-UX B.11.31

Do you have the latest libc patch PHCO_40292, installed?
If so, time to talk to the Response Center.

Re: child process hangs on pthread_mutex_lock

Fedele Giuseppe — Fri, 22 Oct 2010 09:12:34 GMT

The PHCO_40292 patch is NOT present.

Re: child process hangs on pthread_mutex_lock

Fedele Giuseppe — Mon, 25 Oct 2010 07:46:25 GMT

Doet this patch fix this problem or similar issues?

When can I get its delivery notes doc?

Giuseppe

Re: child process hangs on pthread_mutex_lock

Dennis Handly — Mon, 25 Oct 2010 08:14:58 GMT

>Does this patch fix this problem or similar issues?

I just mentioned it because it is the latest. What libc patch do you have?

>When can I get its delivery notes doc?

The same place you get the patch.

Re: child process hangs on pthread_mutex_lock

Fedele Giuseppe — Tue, 26 Oct 2010 14:47:50 GMT

How can I get such info?

Giuseppe

Re: child process hangs on pthread_mutex_lock

James R. Ferguson — Tue, 26 Oct 2010 15:48:02 GMT

Hi Giuseppe:

> When can I get its delivery notes doc?

If you mean more information about the patch Dennis suggested (PHCO_40292) then you can see the patch notes with the patch:

http://www11.itrc.hp.com/service/patch/patchDetail.do?patchid=PHCO_40292&sel={hpux:11.31,}&BC=main|pdb|search|

You can do this by going to the ITRC Patch Database and choosing "find a specific patch" with PHCO_40292 as the argument.

Regards!

...JRF...

Re: child process hangs on pthread_mutex_lock

Fedele Giuseppe — Wed, 27 Oct 2010 10:01:00 GMT

Hello,

I meant the libc patch version: how can I understand this?

In any case thanks for the info.

Giuseppe

Re: child process hangs on pthread_mutex_lock

Dennis Handly — Wed, 27 Oct 2010 10:47:36 GMT

>I meant the libc patch version: how can I understand this?

Your current version can be found with:
swlist -l file | fgrep /usr/lib/hpux32/libc.so.1
OS-Core.CORE2-SHLIBS: /usr/lib/hpux32/libc.so.1
PHCO_39526.CORE2-SHLIBS: /usr/lib/hpux32/libc.so.1

This says I have PHCO_39526 installed.

Re: child process hangs on pthread_mutex_lock

Dennis Handly — Wed, 27 Oct 2010 10:49:30 GMT

You can get a patch family tree with:
http://www.itrc.hp.com/service/patch/patchTree.do?patchid=PHCO_39526