application dumping core during system reboot

[Srimalik]

Hi,

We have an application on 11.23 which dumps core when it is started during system reboot by rc scripts.

The core file shows.

Program terminated with signal 10, Bus error.

warning: The shared libraries were not privately mapped; setting a
breakpoint in a shared library will not work until you rerun the program.

(no debugging symbols found)...(no debugging symbols found)...#0 0xc002f65c in stat64+0xbfffd8f4 () from /usr/lib/dld.sl
(gdb) bt
#0 0xc002f65c in stat64+0xbfffd8f4 () from /usr/lib/dld.sl
warning: Attempting to unwind past bad PC 0xc002f65c
#1 0xc002f658 in stat64+0xbfffd8f0 () from /usr/lib/dld.sl
#2 0xc002f658 in stat64+0xbfffd8f0 () from /usr/lib/dld.sl
(gdb)


interesting thing is that starting the application after system reboot succeeds.

Any clues?

Sri

[Dennis Handly]

This stack trace isn't helpful because of the bogus offsets. What gdb version are you using?

[F Verschuren]

Because you can startup after the reboot manualy the problem can be caused by 2 problems:
1 you started it up to ealy in the boot proces
2 the script was built thet it can only be started manaly, if so you properly alsow can not start it by cron...
reason normaly is that the script does not start whit someling like #!/sbin/sh

[Srimalik]

####gdb version is 3.1

ccxrthp1# /tmp/gdb/gdb -v
HP gdb 3.1 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
####
gdb is showing a proper stack if I start the application manually after system reboot by the same command and force it to dump a core by " kill -ABRT 12345" command.

####

The same application is started by rc scripts without problems on another machine.
This is the first time we are seeing this issue.

Also, our application changeds its CWD after starting and if it dumps, the core should be present in that directory. It has always been the case if the application dumps a core due to some other reasons.

In this case its the core file is created on root dir. So, it make me to think that it dumps core before it actually starts executinng.

Thanks
Sri

[Dennis Handly]

>gdb version is 3.1

The latest is 5.7. You need to download the latest. http://www.hp.com/go/wdb

>gdb is showing a proper stack if I start the application manually

But you don't want to debug that one. :-)
What shlibs does it use?

>it make me to think that it dumps core before it actually starts executing.

That's possible. If there was a dld problem, you would expect a message on stderr. And /usr/lib is mounted.

[Srimalik]

With the new gdb, I am getting a proper stack trace.

################
Program terminated with signal 10, Bus error.

(no debugging symbols found)...#0 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
(gdb) bt
#0 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
#1 0xc001ae4c in map_shlib+0x11ac () from /usr/lib/dld.sl
#2 0xc0018a78 in form_load_graph+0x1a4 () from /usr/lib/dld.sl
#3 0xc001958c in form_load_graph+0xcb8 () from /usr/lib/dld.sl
#4 0xc0028020 in finish_dld_main+0x1024 () from /usr/lib/dld.sl
#5 0xc002b9d4 in _dld_main+0x1c8 () from /usr/lib/dld.sl
#6 0xba8c in __map_dld+0x4e4 ()
#7 0xb0cc in $START$+0xd4 ()
#8 0xc002f658 in get_origin+0x3c () from /usr/lib/dld.sl
(gdb) infor threads
Undefined command: "infor". Try "help".
(gdb) info threads
* 1 system thread 2263 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
(gdb)
############

But still its failing before it enters in our code.

########

I added a trace to find whether /usr/lib is mounted before the command is run...but everything seems to be OK.

Any ideas what may be happening ?

Regards
Sri

[Dennis Handly]

>With the new gdb, I am getting a proper stack trace.

Right, much better.

#0 0xc002f65c in get_origin+0x40 /usr/lib/dld.sl

Do you use $ORIGIN in your shlib paths?

>I added a trace to find whether /usr/lib is mounted before the command is run...but everything seems to be OK.

Well /usr/lib/dld.sl is mounted. What does ldd or chatr show on your executable?

>Any ideas what may be happening?

What version of dld.sl do you have? Maybe you need a patch?
JAGag07378 $ORIGIN in filename cause dlgetfileinfo to dump core

[Srimalik]

>Do you use $ORIGIN in your shlib paths?

We do not use ORIGIN in SHLIB_PATH, but we are using it in embedded path.

>Well /usr/lib/dld.sl is mounted. What does >ldd or chatr show on your executable?
chatr shows: ( please let me know if you want full output)

SHLIB_PATH enabled second
embedded path enabled first
and every path is prefixed with $ORIGIN

ldd resolves all the dependencies without problems.

>What version of dld.sl do you have? Maybe >you need a patch?
>JAGag07378 $ORIGIN in filename cause >dlgetfileinfo to dump core

1# what /usr/lib/dld.sl
/usr/lib/dld.sl:
SMART_BIND
92453-07 dld dld dld.sl B.11.62 070917


This JAG is fixed in patch PHSS_37201, I have already installed this patch but it was of no help. :(


Thanks
Sri

[Srimalik]

the app is starting without problems during reboot on another machine, and dld.sl on that machine seems to be older than that at the machine on which we are facing problems.

ccxrthp2# ls -l /usr/lib/dld.sl
-r-xr-xr-x 1 bin bin 274432 Sep 14 2006 /usr/lib/dld.sl
ccxrthp2# what /usr/lib/dld.sl
/usr/lib/dld.sl:
SMART_BIND
92453-07 dld dld dld.sl B.11.57 060914
ccxrthp2#

[Dennis Handly]

>but we are using it in embedded path.

Yes, that's what I meant.

>and every path is prefixed with $ORIGIN

What are those paths and what does ldd show for them? Is this on a file system that isn't mounted until later?
Is there anyway you can run ldd just before you start your application up??

>This CR is fixed in patch PHSS_37201, I have already installed this patch but it was of no help. :(

I hate to think it broke it. :-(