How to analyze the core files created by applications

[Praveen]

Hi all

How to analyze the core files generated by applications? For example if CODA has crashed and customer wants us to analyze and provide the root cause of core file then how are we going to do that? I tried adb that didn't help. q4 is also useless as it can be used only for kernel crash dumps. any help highly appreciated

Thanks
Praveen

[Srimalik]

try using gdb
the command will be

gdb -c

let us assume that /usr/bin/cp dumped a core and the core file is present in current dir then the command will be

"gdb -c core /usr/bin cp"

then you see the stack using bt command to see the function in which the code was executing. dissassemble will give you the assembly equivalent of the program. etc etc

One thing to note here is you will find is difficult to debug core files generated by release binaries(debug flag disabled while compiling) you will see Hex values in place of function names. you need to be an expert at debugging release binaries.

to make the task easier you should do a debug build of the application.

see man pages for gcc/cc/g++ and gdb for more information.

googling will also help a lot.

cheers
Srikrishan

[A. Clay Stephenson]

You need to use a debugger such as gdb. Stack traces are most useful if the application is compiled with the -g option which includes additional debugger information in the object files; in that case, you can pinpoint the offending source code file and line number. The bad news is that if you have to ask, you probably aren't the one who should be doing the debugging.

[Praveen]

Thanks for the replies. Unfortunately the binaries are not compiled with -g option. ( It is release binary and the core file is from the customer and he wants us to analyse the core file.)Also will gdb work on HPUX? wdb is not installed on the system. Where can I get it?

[A. Clay Stephenson]

wdb is simply a graphical front-end for gdb. Normally, these debuggers are supplied as part of a purchased development system such as aCC (The C++, ANSI/C compiler package) but you can also get gdb from:
http://hpux.its.tudelft.nl/hppd/cgi-bin/search?package=on&description=on&term=gdb

Yes, you can still examine a stack trace using gdb; it's just much easier if it were compiled with -g because there is direct source to object code linkage in that case.

You might consider sending a version of your application compiled with -g to the customer and allowing the program to crash again. You would then have a core file that should be trivially easy to analyze.

Another option would be to have your customer start the application under tusc (or attach the application to tusc) and have them send the tusc output to you. That may well provide you with the data needed to understand the fundamental problem.

Still, the bad news is that if you have to ask about this process ... .

[Praveen]

yes you are right. The best thing is to provide the debug bits. tusc wont be of much help. the applications cores
irregularly. Some times it is painful when customers ask for some thing which is difficult to deliver. Any thank you all for your help. Please let me know if you have any resources for further reference

[Dennis Handly]

You do NOT want to use adb. You should use gdb or wdb, depending on your preferences for GUIs. (Keep in mind any suggestions I give will be for gdb.)

(Is this PA or IPF? What language?)

One simple trick is to use gdb's "bt" command to get a stack trace:
$ gdb executable core
(gdb) bt
(gdb) q

Oh, the most important thing about core files is that they are sticky with the machine they are created. If you don't debug them there, you'll have lots of problems!!

To transport core files, you need to use the packcore and unpackcore commands.

Make sure your customer has the latest gdb, 5.6.

>Srikrishan: you will find is difficult to debug core files generated by release binaries

While this is true, you can still get some info, as long as the executable isn't stripped. You can get the function names. And on IPF, you can get source and line info.

>It is release binary and the core file is from the customer and he wants us to analyse the core file.)

I do this all of the time. How many points do you have? :-)
http://forums1.itrc.hp.com/service/forums/helptips.do?#33

>Also will gdb work on HPUX? wdb is not installed on the system. Where can I get it?

>Clay: but you can also get gdb from:

You should really download it from HP for free:
http://www.hp.com/go/wdb

>Please let me know if you have any resources for further reference

I reviewed some debugging core file documents for WDB. They may be up on the above link. (I don't see it yet. I'll send a message to them to see if they have polished copy to give out.)

I do see:
Debugging Dynamic Memory Usage Errors Using HP WDB

[Dennis Handly · Accepted solution]

Debugging Core Files Using HP WDB:

http://h21007.www2.hp.com/portal/download/files/unprot/wdb/wdb_6.2/Corefiles-updated.pdf

[Praveen]

Thanks a lot. That gives the answer...

[Mithun R N]

Hello Praveen,

Can you please let us know how useful the "Core file debugging" white-paper was to you ?

Thanks and regards,
Mithun
HP-WDB team