Capturing memory leak info for Tuxedo servers written in C

[JVanLierde]

Hi,

I'm trying to instrument our Tuxedo servers on Itanium (11.31) for memory issues using aCC +check=all. The plan is that when the servers are shut down we'll have <binaryname>.<date>.leaks files that we can inspect for leaks.

It's not working. When I shut down the servers using the Tuxedo tmshutdown utility, the server core dumps. Looking at the dump, there was a segment fault in an exit handler. Presumably, this was a Tuxedo exit handler, we didn't register one.

I do successfully get .leaks files on exit from non-Tuxedo binaries, so i'm pretty sure I'm doing it right.

I did notice that there was a <binaryname>.<date> file left over after execution. It's a binary file, so I can't read it. But I'm wondering if this file contains leak information. If so, is there a utilility I can use to process it to a readable format?

[Dennis Handly]

 >using aCC +check=all.

If you only want leaks, you don't really need "all".

>Looking at the dump, there was a segment fault in an exit handler. Presumably, this was a Tuxedo exit handler

Do you have a stack trace of the abort?

Note: An exit handler is needed to dump out the leak info.

[SoumitraC]

---

@JVanLierde wrote:

I did notice that there was a <binaryname>.<date> file left over after execution. It's a binary file, so I can't read it. But I'm wondering if this file contains leak information.

---

The leak information emitted by +check is in text.

[JVanLierde · Accepted solution]

In general, I'm interested in all bad memory operations, and I did use +check=all thinking that all checks are good.

So I tried a more methodical approach, starting with "+check=malloc" and worked my way up to "+check=malloc +check=bounds +check=globals +check=stack" with pretty good results. I did get some core dumps when I added "+check=uninit" and so decided to quite while I was ahead. I need to roll out the switches to all servers and see how works out.

I was able to reliably instrument the servers using batch debugging, so I can always fall back to that mode of operation. My hope was to wire the checks into our makefiles so that developers and our daily builds would be always "on".

Here's the dump I got when I turned on check=uninit:

~/cvs/Head/Dental/GUISupport> gdb DenGUISrv core
HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `DenGUISrv'.
Program terminated with signal 6, Aborted.


warning: No unwind information found.
 Skipping this library /usr/lib/hpux64/libcl.so.1.

#0  0xc0000000002125f0:0 in _lwp_kill+0x30 ()
   from /usr/lib/hpux64/libpthread.so.1
(gdb) bt
#0  0xc0000000002125f0:0 in _lwp_kill+0x30 ()
   from /usr/lib/hpux64/libpthread.so.1
#1  0xc000000000178b70:0 in pthread_kill+0x9d0 ()
   from /usr/lib/hpux64/libpthread.so.1
#2  0xc0000000003f80e0:0 in raise+0xe0 () from /usr/lib/hpux64/libc.so.1
#3  0xc00000001a5a3820:0 in skgesigOSCrash () at skgesig.c:376
#4  0xc00000001b66d0a0:0 in kpeDbgSignalHandler () at kpedbg.c:1074
#5  0xc00000001a5a3cc0:0 in skgesig_sigactionHandler () at skgesig.c:799
#6  <signal handler called>
#7  0xc0000000002125f0:0 in _lwp_kill+0x30 ()
   from /usr/lib/hpux64/libpthread.so.1
#8  0xc000000000178b70:0 in pthread_kill+0x9d0 ()
   from /usr/lib/hpux64/libpthread.so.1
#9  0xc0000000003f80e0:0 in raise+0xe0 () from /usr/lib/hpux64/libc.so.1
#10 0xc000000000509500:0 in abort+0x220 () from /usr/lib/hpux64/libc.so.1
#11 0x4000000000b79290:0 in rtc_at_exit_abort () at rtc_utils.c:298
#12 0xc00000000047c6a0:0 in __exit_handler+0xa0 ()
   from /usr/lib/hpux64/libc.so.1
(gdb)

 rtc_utils.c is not ours, I'm assuming it belongs to Tuxedo.

It looks like I can make it work,it'll just need some tuning.

Thank - JVL

[Dennis Handly]

>I'm interested in all bad memory operations, and I did use +check=all thinking that all checks are good.

It's the cost/benefit that's the worry.  Using globals and stack are expensive.

>worked my way up to "+check=malloc,bounds,globals,stack" with pretty good results.

Using bounds and uninit are helpful.  Using uninit should be very low cost because it isn't instrumented if the compiler knows the variable is initialized.

>I did get some core dumps when I added "+check=uninit" and so decided to quite while I was ahead.

This should be safe.  Most of the work is done when it happens and the exit handler is only there to call abort.

>Here's the dump I got when I turned on check=uninit:

>Skipping this library /usr/lib/hpux64/libcl.so.1.

Unless you are using Fortran, you shouldn't link with this shlib.  Use -lunwind instead.

#11 0x4000000000b79290:0 in rtc_at_exit_abort rtc_utils.c:298

#12 0xc00000000047c6a0:0 in __exit_handler+0xa0

Do you have any uninit messages to stderr before it aborts?

>rtc_utils.c is not ours, I'm assuming it belongs to Tuxedo.

No, that's part of RTC (Run Time Checking).  If you have malloc checking, it would dump out the files.  For uninit, it will call abort if any uninit variables.

And in your case it is working correctly, you have some uninitialized variables and it is aborting.

>It looks like I can make it work, it'll just need some tuning.

Not tuning, you need to initialize those variables.