Analyse a core dump generated very frequently

[Gousset]

Many core dumps have been generated recently by our application which is on a 9000/800/N4000-75, HP-UX rmsph001 B.11.11 U 9000/800 633309331 unlimited-user license server. Could someone have a look at the core dump file and give me any clues as to what the problem could be. File can be downloaded from
http://www.zen56558.zen.co.uk/

[Steven E. Protter]

Shalom Gousset,

There are procedures to help you with the dump.

For example strings

Its also possible to use adb to help work with the data.

You don't mention the name and type application. if its custom written, the developers need to help. If its purchased the vendor should do the work.

SEP

[melvyn burnard]

As already stated, you would be better to get the application vendor to look at this, otherwise if it is a "home-grown" application you would need to get your development team (or person) involved.

[Gousset]

Thanks for the information. From the analysis i can do i have found the following information:

file core
core: core file from 'java'

strings core | grep -i fatal
3FATAL: an unexpected error has occured, desciption:
FATAL: exiting...
TRACE_FATAL
LEVEL_FATAL
traceFatal
FATAL
ORA-17416=FATAL
FATAL
fatal
3FATAL: an unexpected error has occured, desciption:
FATAL: exiting...
TRACE_FATAL
LEVEL_FATAL
traceFatal
FATAL_ERROR_OCCURRED
Fatal Error occured
fatalErrorOccurred
isFatalConnectionError
m_fatalErrorCodes
$Fatal: Resource for javac is missing
fatal.error
fatal.exception
fatalError
-non fatal exception caught in parsing process
fContinueAfterFatalError
getContinueAfterFatalError
isFatal
setContinueAfterFatalError

Could any of the above information help me ? I noticed that there was a ORA-17416, which is an TTC error returned by JDBC drivers.

[A. Clay Stephenson]

The file command is only useful for identifying the actual application that crashed. The strings commands is utterly useless because all you are seeing are the strings associated with potential errors plus any other string constants (e.g. "Enter your Name") embedded in the code. The usual techniques (e.g. a stack trace) are not not terribly useful in this context because you are really seeing an error in the JVM (ie a p-engine) so that you are doubly removed from the actual (Java) source code. By all means, try to fix the Oracle error and also look for and apply the latest Java patches.

[Matti_Kurkela]

I must disagree with SEP here.

A core dump contains the total memory allocation of a program at the point of crash, including all of the program code.

The command "strings core" will display all ASCII strings inside the core file... including _all_ error messages that the program is capable of displaying.

Simply using "strings core" will NOT tell you much of anything useful about the state of the program at the time of crash. It may allow you to identify the crashed program, but that's about all.

Extracting a list of error messages from a core file using "strings" and submitting it to the application support people is going to confuse the heck out of the app support guys... or gives them a good laugh at your expense when you describe how you got that list.

A better way to begin analysis of an unknown core file is to type "file core". Most modern unixes (including HP-UX) will tell you the name of the executable that caused the core dump.

In this case, "file core" reported this:
core: core file from 'java'

To analyze this using gdb, you'd need to have both the core file and the "java" executable (the exact version that produced the core).
The command to do this would be something like:
gdb /opt/java/bin/java ./core

After loading the core file, gdb may report the cause of the program's termination - in this case, it seems to be "Program terminated with signal 6, Aborted."

Further information would be available using gdb's command "bt" - or in the case of multi-threaded programs, "thread apply all bt". This will hopefully list the name(s) of the function(s) or method(s) the program was running at the moment of crash. If the executable does not contain debug information, all you get is a series of memory addresses. Without the source code of the executable *and* symbol tables generated during the compilation of that executable, this won't help much.

My recommendation: First check that your Java version is up to date. If your application requires Java 1.4, try it with Java 1.4.2.. Find the hpjconfig tool from http://www.hp.com/go/java and use it to check that your kernel parametres and OS patches are in order for Java.

If the problem persists with an up-to-date Java version, open a software support case with HP and let them analyze the core file.

If the application is using a commercial J2EE application server platform (like BEA Weblogic or Oracle Application Server), you may wish to try their support options first.