Debugging shared libraries (Apache DSOs) with WDB

[Morgan Pyne]

Hi,

I have built a custom Apache module as a Dynamic Shared Object (DSO) and I am have having problems being able to debug this module.

My Environment is the following:
HP-UX 11.00 on a C3600 machine,
HP ANSIC compiler ( HP92453-01 A.11.01.00)
WDB 2.1 (gdb 5.0)
Apache 1.3.14 (DSO support enabled)

I have naturally built my module with full symbolic debugging information.

Both Apache and my module compile, link and run fine, but I cannot seem to be able to source debug the module.

I have tried a couple of methods:

1. Attaching to a running Apache process, loading the source for my module into wdb, and setting a breakpoint. At this point I am told that a deferred breakpoint is set and will be enabled when the shared library gets loaded. It never gets enabled. I have tried trapping the load of my library with a 'catch load' statement and then re-enabling the breakpoint manually, but this doesn't work either. Although it claims to be re-enabled, it never actually stops (althought it is plain that my code *has* executed)


After trawling the web a little, I found that there might be some issues with setting breakpoints in shared libraries (since other programs need them too). So I tried...


2. Starting Apache directly from within wdb, (so that the shared libraries were privately mapped), and starting Apache with a '-X' parameter i.e. single process mode, to simplify the issue further. I then tried again, both with deferred breakpoints and also using 'catch load' + setting breakpoints, but it seems that these breakpoints are never honoured.

Does any body have any suggestions as to what I am doing wrong or know of any steps that will allow me to debug my module?

Thanks,
Morgan Pyne
SAS EMEA Webmaster

[Rick Beldin]

Make sure that you pre-process your binary to allow for private loading of shared libraries:

/opt/langtools/bin/pxdb -s on binary

This is necessary whenever you want to attach to a binary and set a break point in a shared library. This is a technique I use when I suspect a program is encountering some condition and just calling exit. I'll set a breakpoint on exit and then do a stack trace.

Other tips:
- debug binary not necessary,
but helpful
- symbols necessary

[Morgan Pyne]

Hi Rick,

Thanks for the tip, I hadn't run psdb on the Apache binary. I thought this might do the trick, but unfortunately it has caused another problem.

pxdb seems to kill my binary. When the Apache binary has been processed by it, it seems to hang. It does not reply to any requests on port 80 anymore (indeed I don't think it even binds to the port - I get a 'connection refused' attempting to connect) The behaviour is exhibited both inside and outside wdb. My next step will be to build a debug version of Apache and try to find out where/why it is hanging.

However, since you have experience using pxdb I was wondering if you have seen this behaviour before and if you had any suggested causes/solutions?

Thanks,
Morgan

[Rick Beldin]

No, that sounds very odd. Signals do get rather confusing though when you have attached to the process, because the process that should get the signal is not the process that has attached, but rather the one you have attached to. This can cause you, for example, to have to send signals to an app that would normally get them from the shell.

Does Apache pay attention to SIGHUP or some other signal to know if someone is 'poking' at it? That might be where things are going wrong.

[Morgan Pyne]

Hi Rick,

It's working. I did a lot of tests regarding signals as you suggested above and got very varied results, but I'll spare these details here and instead explain the steps I took to get it working (in case it helps somebody else)

Firstly, I checked to see if there were any patches available which I didn't have installed and which may be relevant. I found PHS_22514, which is an update to the dynamic load library (dld.so) and applied this. Interestingly enough, this patch also seems to provide the dl_open et al. functions to 32bit applications - (whereas I think previously 32bit apps used the shl_xxxx function set, and the dl_xxxx functions were only available for 64bit apps??)

I also decided to use the latest apache sources
(1.3.19) - don't know how relevant this was to solving my problem but I thought I'd mention it anyway. I rebuilt Apache from scratch, and also all of my modules. The only module I built with debugging info was the module I wanted to test.

Next, I ran 'pxdb -s on httpd' on my apache binary to enable private shared library loading, as you had suggested.

I was then able to start apache in single-process mode from within WDB, set a deferred breakpoint in my module, and the debugger stopped as expected when my code was encountered.

So - problem solved, and thanks a lot for your help.

Now, all I have to do is figure out where the stray pointer is in my module, but I think I can manage that by myself :-)

Cheers,
Morgan