Operating System - Linux
1748234 Members
3298 Online
108759 Solutions
New Discussion юеВ

Spawned processes hanging

 
Olu Ogun
New Member

Spawned processes hanging

I work at an ISV and I've got an strange problem at a customer site. We have a server process that spawning sub-processes using fork()/execvp() calls to run child processes and we've noticed on a customer site running HPUX 11.11 that the server intermittently forks (creates child processes) but the execvp() call is never executed. The resulting processes either hang and consume 100% (or 0%) CPU.

The fact that the spawned processes show up in the ps output as the server process means that the execvp code is never run.

The fork/execvp combination works on other UNIXes (Solaris/Lunix/...) w/o problems.

I looked up the HPUX patch DB and came across following issue
(( SR:8606425727 CR:JAGaf85235 )
A multi-threaded process may hang forever trying to fork a new process, if other threads in the process are in kernel context and are holding kernel resources. )

But the above problem is observed on servers which have the patch already applied.

Is doing fork/execvp just flaky on HPUX or is there a know workaround for this issue?

Thanks,

Olu
7 REPLIES 7
Steven E. Protter
Exalted Contributor

Re: Spawned processes hanging

Shalom,

You should be testing software on systems with the latest Gold pack installed, December 2007 because that is the standard in many shops.

No fork is not flaky its probably your code or the OS not being patched.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Dennis Handly
Acclaimed Contributor

Re: Spawned processes hanging

>SEP: No fork is not flaky its probably your code or the OS not being patched.

Right. As long as you do very minimal things between the fork and exec it should work.

Minimal means only open(2)/close(2) and no tricky C++ stuff.
Olu Ogun
New Member

Re: Spawned processes hanging

I was able to catch one of the hanging processes and got the following stack trace

#0 0xc020b2d8 in __ksleep+0x10 () from /usr/lib/libc.2
#1 0xc003df88 in pthread_mutex_lock+0x16c () from /usr/lib/libpthread.1
#2 0xc021cdf8 in __thread_mutex_lock+0x70 () from /usr/lib/libc.2
#3 0xc0197fc4 in _sigfillset+0xc24 () from /usr/lib/libc.2
#4 0xc019aefc in free+0x184 () from /usr/lib/libc.2
#5 0xc00385ec in __specific_data_dealloc+0x24 () from /usr/lib/libpthread.1
#6 0xc003acd4 in __pthread_free+0x37c () from /usr/lib/libpthread.1
#7 0xc003abf8 in __pthread_free+0x2a0 () from /usr/lib/libpthread.1
#8 0xc003bf24 in __pthread_fork_child_cleanup+0xc8 () from /usr/lib/libpthread.1
#9 0xc003efcc in __pthread_fork+0x10c () from /usr/lib/libpthread.1
#10 0xc02161e8 in fork+0x68 () from /usr/lib/libc.2
#11 0xc96bb248 in Lum::ProcessManager::platformSpawn+0x238 () from /u02/testolu/ADB6a/lib/libLMCommon2.sl


It seems to be deadlocked inside the fork() call. I know there are a number of patches said to address this issue but does anyone know the exact patch#. Thanks..

Olu
Dennis Handly
Acclaimed Contributor

Re: Spawned processes hanging

>It seems to be deadlocked inside the fork() call.

Can you go through your other threads and see which are in malloc or free and have the thread locked?

Your CR JAGaf85235 seems related to some kernel issue. This one may be libc & libpthread. I would just install the latest libc and libpthread patches.


Olu Ogun
New Member

Re: Spawned processes hanging

Thanks, the problem is that being an ISV, our customers are very reluctant to patch their systems and tend to want to install the least (or exact) patch(es) that we know fix the issue. I'm trying to open a case with HP Support to get that information..

Olu
Olu Ogun
New Member

Re: Spawned processes hanging

Looking at the server processes, I see the main process still running but with two hanging sub-processes. It's seems that the parent process spawns two process at the same time and one process blocks on a mutex that is used and deleted by the other process. So one runs along but the other hangs forever.. Just my guess..

Olu
Dennis Handly
Acclaimed Contributor

Re: Spawned processes hanging

>the parent process spawns two process at the same time and one process blocks on a mutex that is used

The way it could hang is that one thread has the mutex. The other thread does a fork.

In the child, it is suppose to unlock all locked resources, __pthread_fork_child_cleanup.
But it seems that to free resources, it has to call free and that was the mutex that was locked.