topic Re: program dumps core in Operating System - HP-UX

program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 06:55:26 GMT

Hi @,

Now I am in need of some help:

We do start a program and it dumps core.
# file core
core: core file from 'dmscraper' - received SIGABRT

# what core
core:
HP DCE/9000 1.5 PHSS_27258-59 Module: libcma.1 (Export) Date: Jun 12 2002 12:26:42
PATCH-PHCO_25640 for 10.20; for 10.30, 11.x compatibility libc.1_ID@@/main/r10dav/libc_dav/libc_dav_cpe//1
/ux/core/libs/libc/shared_pa1/libc.1_ID
Nov 30 2001 06:35:42
SMART_BIND
92453-07 dld dld dld.sl B.11.33 020617

Also found: cma_dump.log

Messages in this file:

%Internal DCE Threads problem (version CMA BL10+), terminating execution.
% Reason: cma__open_general: unexpected fstat error
The current thread is 1 (address 0x7b014e78)
DECthreads scheduling database is locked.

Does anybody have a clue why this is happening, and how to troubleshoot this further.

Possible patching issue ? for example: PHSS_28386 ?

I could not relate this patch directly to the issue !

Kind regards for any help,
Clemens

Re: program dumps core

Bruno Vidal — Fri, 27 Jun 2003 07:02:43 GMT

Hi,
What I can suggest is using tusc to know when this signal is caught (which syscall). But you're right, try also to install latest patche related to libs that this process is using (libc, DCE). But it seems really related to a DCE problem, or a locking problem (do you have any messages in dmesg ??).

Cheers.

Re: program dumps core

Robert-Jan Goossens — Fri, 27 Jun 2003 07:11:11 GMT

Clemens,

Hmmmm not sure this is going to help you, it's an oldy.

Date: 5/9/97
Document description: Internal DCE Threads problem - cma__ts_open fd is too large
Document id: W3592711

Robert-Jan.

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 07:11:46 GMT

Hi,

No messages in dmesg ! Too bad :)
This is our production system, so patching is a bit of a problem, unless I can really point to a issue solved with this patch.

Clemens

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 07:14:05 GMT

Robert,

Found that one as well, but was not able to relate this one to the current problem.

Clemens

Re: program dumps core

Alex Glennie — Fri, 27 Jun 2003 07:18:30 GMT

Hi, couldn't tell me what the O/S is here ... I'm a bit confused as there's mention of patches for 11.11 and also mention of DCE 1.5 which is 10.20.

I've seen that error before and it was cured by a patch at 11.00 s700_800 11.00 HP DCE/9000 1.7 Runtime cumulative patch PHSS_27962

the defect being :

7. JAGad27398 : CMA dumps core as it doesn't check fo condition EINPROGRESS.

So getting the latest appropriate dce patch installed may well be a good idea along with libC.

other than than a stack trace gdb ./> ./core may help those more familiar with dce/threads programming something I've not yet got my head around.

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 07:24:38 GMT

Alex,

The os is: B.11.11 U

Clemens

Re: program dumps core

Massimo Bianchi — Fri, 27 Jun 2003 07:32:39 GMT

Hi,
what MC/SG release do you have ?

I saw some defects related also to the combination of MC/SG and DCE kernel thread.

They should be addressed by the patch you mentioned, but i wuold like to know your release.

They also mentioned some test against the cluster using nmap. Did you used that ?

HTH,
Massimo

Re: program dumps core

Massimo Bianchi — Fri, 27 Jun 2003 07:35:52 GMT

Hi,
for example

PHSS_28849: HANG
After an upgrade of ServiceGuard cluster from
version A.10.06 or earlier to A.11.13, cmrunnode
command may fail.

If a port-scanning utility such as the Linux
application "nmap" is executed against a node
running ServiceGuard, cmcld on the node may hang
and unexpectedly fail.

HTH,
Massimo

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 07:37:27 GMT

Massimo,

Sorry it's not related to MC/SG anyway, I posted this by accident in this group.

By the way we are using MC/SG A.11.14

Thanks,
Clemens

Re: program dumps core

Alex Glennie — Fri, 27 Jun 2003 07:44:18 GMT

Clemens,

Not found any info pertinent to the earlier named defect @ 11.11 but dce patches don't always refer to all the fixes they cover in the patch text. My advise would be since this is a live production box you

a) get the stack trace out and either raise a call with HP support or post outputs here.

b) if patches, libc & DCE and DCE client patches can be installed it may be a good starting point.

One other question : has this just started occuring and are any other systems / O/S affected ...

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 07:58:03 GMT

Alex,

I would have tried gdb earlier, but gdb is not on this system, or any other sytem within this environment.
Could someone at HP analyse the core file with gdb? Or do we have to do this on this system ?
If so I will log a call at HP as well.

If the problem just started: no, but I just got started with this problem, since no one else have been able tosolve this, and I just started working with this company.

Patches, might be, but as already stated a bit of a issue right now.

Thx,
Clemens

Re: program dumps core

Alex Glennie — Fri, 27 Jun 2003 08:02:56 GMT

HP support should be able to help with this BUT to analyse the core they'll need a system with the same patch level and access to the binary/application dumping core : it would be quicker/easier if you can install gdb locally and provide them with the stack.

appologies for not being able to help further at this point.

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 08:06:52 GMT

Alex,

I already thought so !:(

Is gdb on the application or OS cd's ?
I thought also there is a difference between 32 bits and 64 bits !

I need 64 bits.

Clemens

Re: program dumps core

Alex Glennie — Fri, 27 Jun 2003 08:13:31 GMT

http://h21007.www2.hp.com/dspp/tech/tech_TechSoftwareDetailPage_IDX/1,1703,1662,00.html

Will have the latest version ... I beleive the apps cd's have slightly older versions .... no need to worry about 32/64 bit ... it will work.

Re: program dumps core

Massimo Bianchi — Fri, 27 Jun 2003 08:17:53 GMT

Hi,
here you can find the gdb:

http://hpux.connect.org.uk/hppd/hpux/Gnu/gdb-5.3/

HTH,
Massimo

Re: program dumps core

Paula J Frazer-Campbell — Fri, 27 Jun 2003 08:51:48 GMT

Re: program dumps core

Clemens van Everdingen — Fri, 27 Jun 2003 10:25:34 GMT

Paula,

Thanks for the very detailed example !

Now I am in desperate need of a slot to install wdb on our production server, and try to find out more.

I will probably not be allowed to do this during production hours.

Too bad but I will have to wait for a maintenance slot. :(

Kind regards,
Clemens

Re: program dumps core

H.Merijn Brand (procura — Fri, 27 Jun 2003 11:10:26 GMT

Depending on how your program was compiled some versions of gdb will help, but some won't: bad luck.

I have installed several versions of the truth, and still cannot tell which version works best on which core dump. Advice: get them all. You don't need reboots for installing wdb/gdb/adb/dde/dbx/xdb

a5:/u/usr/merijn/tmp/itrc 109 > path -al [axgw]db\* dbx ddd dde
2336 100755 -rwx 1 merijn 21156572 16 Dec 2002 23:08 /pro/local/bin/gdb -> ../../../usr/local/pa20_32/bin/gdb
30 100755 -rwx 1 merijn 20864492 16 Dec 2002 23:38 /pro/local/bin/gdb64 -> ../../../usr/local/pa20_64/bin/gdb
70110 100755 -rwx 1 merijn 1612176 27 Apr 1999 12:28 /pro/local/bin/gdb4
201 100555 -r-x 1 bin 196608 10 Nov 1998 03:52 /usr/bin/adb
202 100555 -r-x 1 bin 442368 7 Nov 1997 09:00 /usr/bin/adb64
13716 100555 -r-x 1 bin 4136960 14 Jun 2002 00:58 /opt/langtools/bin/gdb -> ./gdb32
13716 100555 -r-x 1 bin 4136960 14 Jun 2002 00:58 /opt/langtools/bin/gdb32
13717 100555 -r-x 1 bin 4227072 14 Jun 2002 00:58 /opt/langtools/bin/gdb64
13902 100555 -r-x 1 bin 3702784 14 Jun 2002 00:59 /opt/langtools/bin/wdb
30 100755 -rwx 1 merijn 20864492 16 Dec 2002 23:38 /usr/local/pa20_64/bin/gdb
13186 100555 -r-x 1 bin 1003 14 Jun 2002 00:57 /opt/langtools/bin/dde
a5:/u/usr/merijn/tmp/itrc 110 >

l1:/pro/pu/lep/4gl 104 > path -al [axgw]db\* dbx ddd dde
168262 100777 -rwx 1 probev 209 30 Jul 1996 17:45 /pro/pu/local/bin/adbgvi
2243 100755 -rwx 1 merijn 20864492 16 Dec 2002 23:38 /pro/local/bin/gdb64 -> ../../../usr/local/pa20_64/bin/gdb
22951 100755 -rwx 1 merijn 1612176 27 Apr 1999 12:28 /pro/local/bin/gdb4
1189 100755 -rwx 1 merijn 21156572 16 Dec 2002 23:08 /pro/local/bin/gdb -> ../../../usr/local/pa20_32/bin/gdb
99038 100777 -rwx 1 merijn 109 31 Aug 2000 14:15 /pro/local/dbman/bin/xdbish
200 100555 -r-x 1 bin 196608 10 Nov 1998 03:52 /usr/bin/adb
201 100555 -r-x 1 bin 442368 7 Nov 1997 09:00 /usr/bin/adb64
3080 100555 -r-x 1 bin 4136960 14 Jun 2002 00:58 /opt/langtools/bin/gdb -> ./gdb32
3080 100555 -r-x 1 bin 4136960 14 Jun 2002 00:58 /opt/langtools/bin/gdb32
3081 100555 -r-x 1 bin 4227072 14 Jun 2002 00:58 /opt/langtools/bin/gdb64
3626 100555 -r-x 1 bin 3702784 14 Jun 2002 00:59 /opt/langtools/bin/wdb
2243 100755 -rwx 1 merijn 20864492 16 Dec 2002 23:38 /wrk/pa20_64/bin/gdb
1116 100555 -r-x 1 bin 1003 14 Jun 2002 00:57 /opt/langtools/bin/dde
l1:/pro/pu/lep/4gl 105 >

FWIW, gdb-5.3 is available from the HP-UX porting center, and in my gcc ports on https://www.beepz.com/personal/merijn/ or http://www.cmve.net/~merijn/

Enjoy, have FUN! H.Merijn

Re: program dumps core

Jean-Louis Phelix — Fri, 27 Jun 2003 11:31:28 GMT

Hi,

You seem to currently have DCE patch 27258. I had a look to its most recent successor (PHSS_28386) and guess what ...

JAGae52205 : A threaded application that links to libcma, some times, exits unexpectedly after the system call accept() has been made.

The accept() call in cma in turn calls fstat() to obtain socket information. Sometimes, fstat()returns an error number, which is not handled in the code, due to which the problem occurs.

Perhaps you could try it ...

Regards.