java core dump analysis

[Fedele Giuseppe]

Hello,

I need to analyse core file generated by a Java application on a remote itanium machine:

file core
core: ELF-32 core file - IA64

I have the core file but I cannot access the remote machine. Moreover I can access to an equivalent machine.

How can I proceed?
Which tool can I use?

Regards
Giuseppe

[Steven E. Protter]

Shalom Guiseppe,

You might get something to at least look at with the strings command.

That might get you something to search by.

As a precaution, I would insure you have recommended java patches and a current version from http://www.hp.com/go/java

SEP

[kobylka]

Hello Fedele!


On HP-UX you can use wdb or adb for debugging.

WDB is HP's port of GDB for HP-UX, get it here:

http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801?ciid=cb080f1bace021100f1bace02110275d6e10RCRD


To load the corefile and see where it crashed and why load the executable that produced the crash and the corefile like this:

gdb /path/to/java /path/to/core

or

adb /path/to/java /path/to/core

or, for graphical frontend to gdb,

wdb /path/to/java /path/to/core


for further information consult manpages (man gdb, man wdb, man adb) and online references:

wdb - http://docs.hp.com/en/oshpux11i.html#Debugging%20Tools

gdb - http://dirac.org/linux/gdb/



Beware that if you move the corefile from one system to another you will need Java installed on the target system too (same executable as on source system is strongly encouraged).


Kind regards,

Kobylka

[Fedele Giuseppe]

Hello,

if I run the command:

# /opt/langtools/bin/gdb64 /opt/java1.5/jre/bin/java /usr/Systems/HA_9.1.0_Master/core

I obtain the following output:

Detected 32-bit executable.
Invoking /usr/ccs/bin/gdb32.
HP gdb 5.9 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00
and target hppa1.1-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.9 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core file is in different format from executable
To unload the executable use the "file" command with no argument

(gdb)

What does the last message means ("Core file is in different format from executable
To unload the executable use the "file" command with no argument") ?

Can I proceed? And in which way?

Giuseppe

[kobylka]

Hello Fedele!

Seems like you are on a PA-RISC based machine:

> HP gdb 5.9 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00


trying to load a corefile from an Itanium executable:

> Core file is in different format from executable


1. First you need to get on an Itanium based machine with gdb/wdb installed.

2. Then you need to find the executable for the corefile. If you are sure the corefile was generated by Java then run gdb with the java executable as you did in the last post (but on Itanium).

Just to be sure the corefile was generated by the java executable try this:

gdb -core /usr/Systems/HA_9.1.0_Master/core

It should print a line like this:

"Core generated by executable_name"


When you found the needed executable it should load fine so you can do a backtrace.

Kind regards,

Kobylka

[Dennis Handly]

> I need to analyse core file generated by a Java

What do you expect to find?

>/opt/langtools/bin/gdb64 /opt/java1.5/jre/bin/java

This is the wrong java. This PA one is the driver and immediately execs the real one in:
/opt/java1.5/jre/bin/IA64N/java
/opt/java1.5/jre/bin/IA64W/java

>Kobylka: Beware that if you move the corefile from one system to another you will need Java installed on the target system too (same executable as on source system is strongly encouraged).

Don't even bother doing this unless you use the packcore command to copy ALL of the files.

[Fedele Giuseppe]

Hi Kobylka, hi Dennis,

first of all I made a mistake: I had installed WDB PA-RISC version on an ITANIUM machine.

Now I have installed WDB 5.9 + patch PHSS_38157 on my ITANIUM machine.

If I try to check my core file as you have suggested, I obtain the following error:

trhpli,sys,root # gdb -core ./core
HP gdb 5.9 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.9 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.


warning: Unknown symbols for './core'; use the 'symbol-file' command.

Couldn't fetch registers from core file: File in wrong format
No object file symbols.
Couldn't fetch registers from core file: File in wrong format
Couldn't fetch registers from core file: File in wrong format

warning: Unable to read the load_info structure address from .
#0 0x0 in ()

How can I proceed?

Thanks

Giuseppe

[kobylka]

Hello Fedele!

Seems gdb doesn't handle -core behaviour on cross debugging.


Ok, basically you need the executable which generated the corefile. Since you cannot access the machine where the corefile is from, but you know it is a java corefile you need to install the correct JDK/JRE in order to get the "java" executable. Do you remember what JDK/JRE version was installed on the machine you can't access?

If yes, install the corresponding version from
https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXJAVAHOME

If not, try this to narrow down the possibilities:

strings -n 3 core | more

and look for a string like:

PATH=...

this could give you the path where the java executable was from if it was executed on command line.

Furthermore, watch out for strings like, for example, ./java or /opt/java1.5/bin/java.


Kind regards,

Kobylka


Dennis: I owe you a response to your post :)

[Fedele Giuseppe]

Hi,

by executing:
strings -n 3 ./core | grep java | grep bin

I get:

/opt/java1.5/jre/bin/IA64N/java

that is the same installed on my machine:

-r-xr-xr-x 1 bin bin 236912 Mar 9 2009 /opt/java1.5/jre/bin/IA64N/java

Giuseppe

[kobylka]

Hi!


Try to load it into gdb:

# gdb /opt/java1.5/jre/bin/IA64N/java /path/to/core



The only problem I see is there are several versions of Java 1.5 (JDK or JRE):

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXJDKJRE50

if your local version doesn't match exactly your corefile's version then there could be errors running gdb.


Kind regards,

Kobylka

[Fedele Giuseppe]

Hi,

I met again the previous error

gdb /opt/java1.5/jre/bin/IA64N/java /usr/Systems/HA_9.1.0_Master/core
HP gdb 5.9 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.9 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..

Couldn't fetch registers from core file: File in wrong format
Couldn't fetch registers from core file: File in wrong format
Couldn't fetch registers from core file: File in wrong format

warning: Unable to read the load_info structure address from .
#0 0x0 in ()


QUESTION: since the remote core dump has lead the file system in a FULL condition, is it possible that the core file has not been correctly produced?

Giuseppe

[kobylka]

> QUESTION: since the remote core dump has lead the file system in a FULL condition, is it possible that the core file has not been correctly produced?


Insufficient disk space can lead to truncated corefiles, indeed.

Check if core is truncated:

# what core


and

# elfdump -o -S core


Please attach the output of both commands.


Kind regards,

Kobylka

[Dennis Handly]

>I had installed WDB PA-RISC version on an Integrity machine.

This is fine, you need it for PA debugging.

>Now I have installed WDB 5.9 + patch PHSS_38157 on my Integrity machine.

Again, this is for PA, you need to download wdb 6.0 from http://www.hp.com/go/wdb

>the file system in a FULL condition, is it possible that the core file has not been correctly produced?

Yes, as Kobylka said. You might as well remove that core file and wait until the next time.
How big was it? You may have to increase the "ulimit -c" value.

>kobylka: basically you need the executable which generated the corefile. Since you cannot access the machine where the corefile is from,

Don't go there. You need more than the executable. If you can't access that machine, you need to give up now.
That's why you use packcore so you you copy ALL of the load modules.

[Fedele Giuseppe]

Hi Kobylka,

please see the commands output:

what core
core:
1.5.0.11-_07_nov_2007_17_04 (java) Built: 07/11/07-17:07 View: jinteg_h1.5.0.11.rc1b1
default.syslogd $Date: 2006/01/23 20:01:10 $Revision: r11.23/2 PATCH_11.23 (PHCO_34253)


trhpli,sys,root # elfdump -o -S core

core:

*** Program Header ***

Type Offset Vaddr FSize Memsz

NULL 00000000 00000000 00000000 00000000
..................... (my comment: 111 lines)
NULL 00000000 00000000 00000000 00000000

[Dennis Handly]

>elfdump -o -S core
>NULL 00000000 00000000 00000000 00000000

Your core file has been truncated, remove it.

[kobylka]

Hello Fedele!


As Dennis already suggested, your corefile is truncated and thus cannot be used to load into a debugger.

Problem might have been your disk filling up when core dump was generated or that your core size limit was too low.

If your Java application erronously filled up the filesystem where it was running maybe you should try to make the application use this filesystem (maybe using absolute paths for creating files or whatever) and chdir to a different filesystem with enough space. This would leave the corefile in the chdir'ed directory rather than on the filled up filesystem (which would most probably truncate the corefile again).


Kind regards,

Kobylka

[Dennis Handly]

>kobylka: This would leave the corefile in the chdir'ed directory

Since Fedele is on 11.31, he can also use the coreadm(1m) command to redirect core files somewhere else.