DDE problem

[Sabine Irmer]

A user of my site reported me this compiling error message :
I'd like to use the DDE-debugger under MPI. But I have
no success to get a working GUI. I'm faced with the problem
already when I try the following simple example:

Edit hello1.f:

program hello1
include 'mpif.h'
integer ierror, rank, comm
call mpi_init(ierror)
call mpi_comm_rank(MPI_comm_world,rank,ierror)
write(*,*) 'hello world, rank=',rank
call mpi_finalize(ierror)
end

mpif90 -o hello1 -g hello1.f
dde hello1 (to test DDE alone)
==> O.K.:
Executable:/work/crh/8g/test/hello1
PC Location: hello1 File: hello1.f Line: 7
Source: /work/crh/8g/test/hello1.f
(To continue debugging this run makes no sense,
because the call mpi_init needs mpirun.)

export MPI_FLAGS=edde
mpirun -np 2 hello1
==> DDE GUI is opened:
Executable:/work/crh/8g/test/hello1
but:
PC Location: `va(0) File: Line: 0
Source: no source for `va(0) [00000000]
Stopped at: `va(0) (00000000) (`thread(1))

Neither in dynatext nor on man pages I could find any hint
to overcome this problem. What's wrong ?
Thanks ! Roland

[Sam Nicholls · Accepted solution]

This is known problem that is not yet documented. By default, MPI processes create a hidden kernel thread to manage direct process-to-process bcopy between ranks. DDE is attaching to the p2p bcopy threads instead of the main thread. Since the p2p bcopy threads are kernel-only, there is no user-space context so dde gets an invalid PC.

The workaround is to disable MPI p2p bcopy by setting the MPI_2BCOPY env var...

$ export MPI_2BCOPY=1

This disables p2p bcopy, so that message transfers must be copied twice, once into shared memory and once out of it. Depending on the nature of the application, disabling p2p bcopy usually decreases performance. But, this shouldn't be a problem for debugging.

This problem/workaround will either be fixed or documented for the next MPI release.

[Sabine Irmer]

Thank you, Sam Nicholls,

for the prompt answer. Sorrily it didn't change the situation essentially.
The output is now:

PC Location: _select_sys+0010 File: Line: 0
Source: no source for _select_sys+0010 [c01f2b68]

Stopped at: _select_sys+0010 (C01F2B68)

What else must we do to get the debugging run ?

[Sam Nicholls]

Hi Sabine,

This is normal behavior. You're almost there. The hello1 processes are stopped deep within the mpi_init() call. That's why dde doesn't show any source. Do the following in both dde windows...

Use the down-arrow stack frame button to back out to the "main" frame. It should be stack frame 8 and your source code will appear.

Next, set a breakpoint somewhere after the mpi_init call. Then, in the debugger input window, set the MPI_DEBUG_CONT global variable to 1...

set MPI_DEBUG_CONT = 1

Now, do press "continue" and the program will proceed to the breakpoint. From here you can debug as you normally would.

What's happening is that, within mpi_init(), each hello1 process detects the MPI_FLAGS=edde environment variable. So, before returning from mpi_init(), they fork/exec dde processes to attach to themselves. They can't return from mpi_init() immediately because they have to give the dde processes a chance to attach. So, the processes spin on the MPI_DEBUG_CONT global variable which is initialized to 0. By setting it to 1 within DDE, you allow the processes to continue out of mpi_init().

The MPI User's Guide talks about this in the "Debugging and Troubleshooting" chapter. But it doesn't mention that you have to backout to the main stack frame to see your source. It's a somewhat convoluted solution. But there is no easy way to support multiprocess MPI programs with single-process debuggers.

There's a really nice debugger for MPI called TotalView. It supports multiprocess and multithread parallel programs (MPI, OpenMP, pthreads). It's available for HP-UX, Solaris, Linux, AIX, and others. And it supports C, C++, Fortran (77 and 90), and HPF. Etnus worked closely with the MPI group at HP to port TotalView to HP-UX. Demo copies of totalview are available for free.

For more info, goto http://www.etnus.com/

-sam

[Sabine Irmer]

Hi Sam,

many thanks for your helpful instructions and explanations to run DDE under MPI.
Our simple example "hello1" is running well now. This is the good news and a
reason to enjoy.

Sorrily our real program to debug is unable to continue from mpi_init in spite
of setting MPI_DEBUG_CONT = 1. The parallel processes are shown as running in
the top table but don't use CPU time. Whithout DDE the same binary file is
running well. Are there any other conditions which are neccessary for mixing MPI
and DDE, maybe a limit of the program file length to debug ? What informations
are interesting to fix this problem ? Do you have an idea ?

You mentioned the wayout "Totalview". Till now we don't have this debugger. But
if you expect more problems with MPI and DDE this would be a strong argument for
us to apply for Totalview.

With best regards
Roland

[Sam Nicholls]

Roland,

The procedure you used for the simple test program should work for your application. It sounds like the ranks are still blocking within mpi_init even though you told them to continue. Here are some things to check...

- Make sure you are setting MPI_DEBUG_CONT = 1 and doing the continue in *each* process. mpi_init() is a collective operation and all the ranks must return from it together. It calls to mpi_barrier() before returning and mpi_barrier() blocks until all ranks have called it.

- Does the application do anything significant before calling mpi_init()? Make it so the app calls mpi_init before doing anything else.

- Install wdb and try it with MPI_FLAGS=egdb to see if the problem is specified to dde.

-sam