core files

[S.S.]

Dear All,

Yesterday, a core file was generated at 18:47 under /opt filesystem.
-rw------- 6467788 /opt/Hewlett-Packard/RSMHostSW/RSMHA/core

Also, yesterday the /opt and /tmp filesystem were filled up to 100% due to the following files generated:

/opt:

-rw-r--r-- 1 root root 136847736 Mar 7 10:39 halogs-100307-103658.tar.gz

-rw-r--r-- 1 root root 266588833 Mar 7 12:42 halogs-100307-123919.tar.gz

-rw-r--r-- 1 root root 2521876480 Mar 7 14:45 halogs-100307-144210.tar

---------- 1 root root 393756672 Mar 7 16:48 halogs-100307-164446.tar

-rw-r--r-- 1 root root 0 Mar 7 18:47 halogs-100307-184703.tar


/tmp:

-rw-r--r-- 1 root root 2564046848 Mar 7 16:47 halogs-100307-164446.tar

May i know how can i debug this core file.

Also suggest me if core files generate what are the steps to do.

My OS version is 11.11

Thanks!

[Johnson Punniyalingam]

>>May i know how can i debug this core file.

Also suggest me if core files generate what are the steps to do<<

if your server under HP Maintenance Support, better advice to approach them :)


Also Check below Thread, for Core file Analysis

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1412710

[Kapil Jha]

are you running veritas cluster on the server.

# what /opt/Hewlett-Packard/RSMHostSW/RSMHA/core

show this output.

Tar files does not look like core can u untar on of the file and run
#what file

BR,
Kapil+

[Dennis Handly]

>May I know how can I debug this core file.

Do you have gdb installed?
gdb executable core
(gdb) bt
(gdb) q

>suggest me if core files generate what are the steps to do.

You need to contact the Response Center to see if there are any fixes.

If you don't want that core file, you can do a "mkdir core" there.

>Johnson: Check below thread, for Core file Analysis

Not hardly. kwdb and q4 are for crash dumps, not application core files. You can download wdb/gdb and core file debugging from:
http://www.hp.com/go/wdb
Debugging core files using HP WDB
file .../core will tell you the signal and the basename of the executable.

[S.S.]

Hi,

The output is:

# what /opt/Hewlett-Packard/RSMHostSW/RSMHA/core
/opt/Hewlett-Packard/RSMHostSW/RSMHA/core:
$Revision: B.11.11$
libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.11.10 /ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Sep 14 2006,17:45:41
HP SNIA HBA API Vendor-fcd Library: libhbaapifcd.sl Sep 14 2006
$Revision: B.11.11$
HP SNIA HBA API Vendor Library: libhbaapihp.sl Sep 4 2006 12:17:27
$ PATCH_11.11/PHCO_28605 Jul 31 2003 03:09:31 $
libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.11.10 /ux/kern/kisu/FCD/src/common/wsio/fcd_init.c:Sep 14 2006,17:45:41
$ B.11.11_LR Oct 26 2000 23:10:23 $
Pthread Interfaces
$Revision: libpthread.1: @(#) depot-32pa R11.11_BL2003_1209_2 PATCH_11.11 PHCO_29960 Tue Dec 9 01:51:37 PST 2003 $
HP-UX libm shared PA1.1 C Math Library 20000331 (201031) UX11.01
HP-UX libisamstub.sl 19991217 (135120) B3907DB/B3909DB B.11.01.11
HP Port of Compaq Convert RTL V0.0.00
HP Fortran of Alpha RT V0.0.00
Intel Fortran RTL V1.1-929 1+ 1-Aug-2003
fs_amod.s $Revision: 1.9.1.1 $
libcl.sl version B.11.XX.21 - Jun 22 2005
$ PATCH_11.11/PHCO_38279 Jul 3 2009 04:56:42 $
SMART_BIND
92453-07 dld dld dld.sl B.11.68 090608



Also,

the gdb is installed on my machine.

thanks!

[S.S.]

I have invoked the wdb and i have received the following error message:

# /opt/langtools/bin/wdb
Error: Can't open display:
Error: Couldn't find per display information


Thanks!

[Dennis Handly]

>I have invoked wdb ... Error: Can't open display:

Stick with the simpler gdb.

[S.S.]

Hi Dennis,

For gdb it has been invoked.

Now, can you suggest me what steps should i execute under gdb.

# gdb
HP gdb 5.4.0 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00
and target hppa1.1-hp-hpux11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.4.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

(gdb)

Thank you.

[Johnson Punniyalingam]

Check Below thread,

Hope this Helps

http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801?ciid=f6ccb52bdb779110VgnVCM100000275d6e10RCRD#gdb_debug

Rgds,
Johnson

[Dennis Handly]

>can you suggest me what steps should I execute under gdb.

As I mentioned above:
gdb path-to-executable path-to-core
(gdb) bt
(gdb) info reg
(gdb) q

>HP gdb 5.4.0

You might want to download a newer version.

>Johnson: Hope this helps

Not sure why you would want to know about Aries debugging on a PA 11.11 system?

[Kapil Jha]

these tar files are created by HP Replication Solutions Manager(RSM)

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3807693&prodTypeId=18964&objectID=c01677859

You may have to check why its generating so much logs and may be restart the rsm agent.

BR,
Kapil+

[S.S.]

Hi,

Today, i have found that the core files are generated in /home filesystem also.
These files are generated every day since 8th March.

-rw------- 1 dba 136197628 Mar 8 19:10 core.24320
-rw------- 1 dba 136197628 Mar 8 19:10 core.10115
-rw------- 1 dba 136197628 Mar 9 19:10 core.6579
-rw------- 1 dba 136197628 Mar 9 19:10 core.22142
-rw------- 1 dba 136197628 Mar 10 19:10 core.22891
-rw------- 1 dba 136197628 Mar 10 19:10 core.8357
-rw------- 1 dba 136197628 Mar 11 19:10 core.10714
-rw------- 1 dba 136197628 Mar 11 19:10 core.26272

Btw, We did the patching on March 7th and installed only the security patches on this server.

I have checked with DBA team, they said that one of the script actually starts at 19:10 and will create a log file in some other location in which that directory path is not existed.
May i know due to that the core dumps are happening?

Thank you.

[Kapil Jha]

All this is amazing, initially I did not see any core file (at least for me) now I have at.

Oracle says
>>>they said that one of the script actually starts at 19:10 and will create a log file in some other location in which that directory path is not existed.

what does it means exactly.

anyways these are real core files and not log files.
core.10115 ----created for process id 10115

what core.10115

would tell you why it created what signal caoused this core file.

BR,
Kapil+

[Kapil Jha]

We do not know what that script do,
you have to check that acript which runs at 19:10

BR,
Kapil+

[S.S.]

Hi,

This was terminated with signal 11.

Checked by using gdb.
(gdb) core core.10115
warning: Unknown symbols for 'core.10115'; use the 'symbol-file' command.
Core was generated by `sh'.
Program terminated with signal 11, Segmentation fault.

#0 0xc00264a4 in ()
Cannot access memory at address 0xc00264a4

and the 'what' output:

# what core.10115
core.10115:
Version M-11/16/88f
$ B.11.11_LR Mar 11 2008 04:52:07 $
$ PATCH_11.11/PHCO_38279 Jul 3 2009 04:56:42 $
SMART_BIND
92453-07 dld dld dld.sl B.11.68 090608

[Laurent Menase]

Check your sh version is the last one.
PHCO_40121

Since you already have the last libc according to the what string, if you have the last sh, then it would make sens to ask to HP support.

[Dennis Handly]

>(gdb) core core.10115
>warning: Unknown symbols for 'core.10115'; use the 'symbol-file' command.
Core was generated by `sh'.

If you use: gdb /usr/bin/sh core.10115
then you can use "bt" to get a stack trace.

>Kapil: what core.10115
would tell you why it created what signal caused this core file.

A better command is "file core.10115".