topic Re: application dumping core during system reboot in Operating System - HP-UX

application dumping core during system reboot

Srimalik — Fri, 26 Oct 2007 03:02:52 GMT

Hi,

We have an application on 11.23 which dumps core when it is started during system reboot by rc scripts.

The core file shows.

Program terminated with signal 10, Bus error.

warning: The shared libraries were not privately mapped; setting a
breakpoint in a shared library will not work until you rerun the program.

(no debugging symbols found)...(no debugging symbols found)...#0 0xc002f65c in stat64+0xbfffd8f4 () from /usr/lib/dld.sl
(gdb) bt
#0 0xc002f65c in stat64+0xbfffd8f4 () from /usr/lib/dld.sl
warning: Attempting to unwind past bad PC 0xc002f65c
#1 0xc002f658 in stat64+0xbfffd8f0 () from /usr/lib/dld.sl
#2 0xc002f658 in stat64+0xbfffd8f0 () from /usr/lib/dld.sl
(gdb)

interesting thing is that starting the application after system reboot succeeds.

Any clues?

Sri

Re: application dumping core during system reboot

Dennis Handly — Fri, 26 Oct 2007 03:19:18 GMT

This stack trace isn't helpful because of the bogus offsets. What gdb version are you using?

Re: application dumping core during system reboot

F Verschuren — Fri, 26 Oct 2007 03:53:50 GMT

Because you can startup after the reboot manualy the problem can be caused by 2 problems:
1 you started it up to ealy in the boot proces
2 the script was built thet it can only be started manaly, if so you properly alsow can not start it by cron...
reason normaly is that the script does not start whit someling like #!/sbin/sh

Re: application dumping core during system reboot

Srimalik — Fri, 26 Oct 2007 04:10:05 GMT

####gdb version is 3.1

ccxrthp1# /tmp/gdb/gdb -v
HP gdb 3.1 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
####
gdb is showing a proper stack if I start the application manually after system reboot by the same command and force it to dump a core by " kill -ABRT 12345" command.

####

The same application is started by rc scripts without problems on another machine.
This is the first time we are seeing this issue.

Also, our application changeds its CWD after starting and if it dumps, the core should be present in that directory. It has always been the case if the application dumps a core due to some other reasons.

In this case its the core file is created on root dir. So, it make me to think that it dumps core before it actually starts executinng.

Thanks
Sri

Re: application dumping core during system reboot

Dennis Handly — Fri, 26 Oct 2007 04:37:14 GMT

>gdb version is 3.1

The latest is 5.7. You need to download the latest. http://www.hp.com/go/wdb

>gdb is showing a proper stack if I start the application manually

But you don't want to debug that one. :-)
What shlibs does it use?

>it make me to think that it dumps core before it actually starts executing.

That's possible. If there was a dld problem, you would expect a message on stderr. And /usr/lib is mounted.

Re: application dumping core during system reboot

Srimalik — Fri, 26 Oct 2007 05:58:04 GMT

With the new gdb, I am getting a proper stack trace.

################
Program terminated with signal 10, Bus error.

(no debugging symbols found)...#0 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
(gdb) bt
#0 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
#1 0xc001ae4c in map_shlib+0x11ac () from /usr/lib/dld.sl
#2 0xc0018a78 in form_load_graph+0x1a4 () from /usr/lib/dld.sl
#3 0xc001958c in form_load_graph+0xcb8 () from /usr/lib/dld.sl
#4 0xc0028020 in finish_dld_main+0x1024 () from /usr/lib/dld.sl
#5 0xc002b9d4 in _dld_main+0x1c8 () from /usr/lib/dld.sl
#6 0xba8c in __map_dld+0x4e4 ()
#7 0xb0cc in $START$+0xd4 ()
#8 0xc002f658 in get_origin+0x3c () from /usr/lib/dld.sl
(gdb) infor threads
Undefined command: "infor". Try "help".
(gdb) info threads
* 1 system thread 2263 0xc002f65c in get_origin+0x40 () from /usr/lib/dld.sl
(gdb)
############

But still its failing before it enters in our code.

########

I added a trace to find whether /usr/lib is mounted before the command is run...but everything seems to be OK.

Any ideas what may be happening ?

Regards
Sri

Re: application dumping core during system reboot

Dennis Handly — Fri, 26 Oct 2007 17:32:03 GMT

>With the new gdb, I am getting a proper stack trace.

Right, much better.

#0 0xc002f65c in get_origin+0x40 /usr/lib/dld.sl

Do you use $ORIGIN in your shlib paths?

>I added a trace to find whether /usr/lib is mounted before the command is run...but everything seems to be OK.

Well /usr/lib/dld.sl is mounted. What does ldd or chatr show on your executable?

>Any ideas what may be happening?

What version of dld.sl do you have? Maybe you need a patch?
JAGag07378 $ORIGIN in filename cause dlgetfileinfo to dump core

Re: application dumping core during system reboot

Srimalik — Mon, 29 Oct 2007 01:33:30 GMT

>Do you use $ORIGIN in your shlib paths?

We do not use ORIGIN in SHLIB_PATH, but we are using it in embedded path.

>Well /usr/lib/dld.sl is mounted. What does >ldd or chatr show on your executable?
chatr shows: ( please let me know if you want full output)

SHLIB_PATH enabled second
embedded path enabled first
and every path is prefixed with $ORIGIN

ldd resolves all the dependencies without problems.

>What version of dld.sl do you have? Maybe >you need a patch?
>JAGag07378 $ORIGIN in filename cause >dlgetfileinfo to dump core

1# what /usr/lib/dld.sl
/usr/lib/dld.sl:
SMART_BIND
92453-07 dld dld dld.sl B.11.62 070917

This JAG is fixed in patch PHSS_37201, I have already installed this patch but it was of no help. :(

Thanks
Sri

Re: application dumping core during system reboot

Srimalik — Mon, 29 Oct 2007 02:11:25 GMT

the app is starting without problems during reboot on another machine, and dld.sl on that machine seems to be older than that at the machine on which we are facing problems.

ccxrthp2# ls -l /usr/lib/dld.sl
-r-xr-xr-x 1 bin bin 274432 Sep 14 2006 /usr/lib/dld.sl
ccxrthp2# what /usr/lib/dld.sl
/usr/lib/dld.sl:
SMART_BIND
92453-07 dld dld dld.sl B.11.57 060914
ccxrthp2#

Re: application dumping core during system reboot

Dennis Handly — Mon, 29 Oct 2007 05:27:18 GMT

>but we are using it in embedded path.

Yes, that's what I meant.

>and every path is prefixed with $ORIGIN

What are those paths and what does ldd show for them? Is this on a file system that isn't mounted until later?
Is there anyway you can run ldd just before you start your application up??

>This CR is fixed in patch PHSS_37201, I have already installed this patch but it was of no help. :(

I hate to think it broke it. :-(

Re: application dumping core during system reboot

Srimalik — Mon, 29 Oct 2007 08:17:09 GMT

>What are those paths
see chatr.txt in attached gz file

> and what does ldd show for them?

ldd_after_start.txt in attcahed gz

>Is this on a file system that isn't mounted >until later?
>Is there anyway you can run ldd just before >you start your application up??

see info_during_reboot.txt in attached gz

Sri

Re: application dumping core during system reboot

Srimalik — Thu, 01 Nov 2007 04:39:46 GMT

Hi, Dennis

Was the data useful ?

Thanks
Sri

Re: application dumping core during system reboot

Dennis Handly — Sat, 03 Nov 2007 02:47:28 GMT

>Was the data useful?

Nothing obvious.
If you still have your core file can you do:
(gdb) x /s *(void**)($sp-0x40)
(gdb) x /s *(void**)($sp-0x3c)
(gdb) p /x $ret0

This should print out the input string to get_origin. And the result of dld_realpath and the result of dld_strrchr.

In your chatr(1) output I see:
dynamic /usr/lib/libstd.2
dynamic /usr/lib/libstream.2
dynamic /usr/lib/libCsup.2

But some of the ldd output has the wrong order:
/usr/lib/libCsup.2 => /usr/lib/libCsup.2
/usr/lib/libstream.2 => /usr/lib/libstream.2
/usr/lib/libstd.2 => /usr/lib/libstd.2

You may have to use "ldd -v" or chatr(1) on every shlib until you find this bad order.
(Or perhaps ldd doesn't have any ordering in its output??)

I do see some strange relative paths:
/usr/lib/libdld.2 => ../lib/libdld.2

But this is in the "after" list too.
Comparing the RHS of each "=>" I get matches for before and after.

Your files that use $ORIGIN all appear to be in /opt/VRTSob, which appears to be:
/opt on /dev/vg00/lvol5 ioerror=mwdisable,largefiles,delaylog,dev=40000005

I did find out that one of your shlibs is illegally being built with -AA. You can't mix -AA and -AP. You have one with libCsup_v2.2 and the rest with libCsup.2.

Re: application dumping core during system reboot

rajdev — Sat, 03 Nov 2007 10:40:02 GMT

Hi,

Not sure what the issue is but since you say that its working after system is booted i have a couple of questions :
---> how are you starting it ie from command prompt
---> does this require any terminal ( what is the stdin/stdout/stderr )
---> or is this a daemon program
---> have you tried running with nohup

Regards,
RD

Re: application dumping core during system reboot

Srimalik — Sat, 03 Nov 2007 11:31:20 GMT

Thanks, Dennis

Output of gdb commands given by you:
(gdb) x /s *(void**)($sp-0x40)
0x77ff0000: "vxsvc"
(gdb) x /s *(void**)($sp-0x3c)
0x77fce36c: ""
(gdb) p /x $ret0
$1 = 0x0
(gdb)

I can not make out much from the output. :(

I would be able to work on all other points on Monday only. :(

Rajdev,
This exe does not use teminal for stdot/in. We don't have to use nohup.
The command used is exactly same as the command used after reboot and as I have mentioned earlier, it works without problem on most of the machines.

Both of you have a great weekend. :)

Re: application dumping core during system reboot

Dennis Handly — Sat, 03 Nov 2007 16:53:25 GMT

>Output of gdb commands given by you:
(gdb) x /s *(void**)($sp-0x40)
0x77ff0000: "vxsvc"
(gdb) x /s *(void**)($sp-0x3c)
0x77fce36c: ""

Basically this is saying that argv[0]? is not an absolute path so that dld's realpath has to sweat. And something is going wrong there.
And unfortunately dld doesn't have any checking for this case.

Years ago I found problem using dirname/basename on argv[0] and I had a case of a non-absolute path and the function aborted. Later I was never able to duplicate it because ksh always provided the absolute path every time I tried to duplicate it. Perhaps that's happening to you with the before and after the boot? Something to do with shells and exec(2)?

So the workaround may be simple, provide an absolute path to vxsvc. (I assume that's what you were running?)

In the meantime, you should contact the Response Center with your problem so they can put more checking into dld. Unfortunately if realpath doesn't work or getcwd(3) fails, dld may give you an error that it can't compute $ORIGIN and you'll just abort with a nicer message.

(You're not in a directory that gets removed out from under it?)

(It would have been helpful if your attachment was .tar.gz or .tgz instead of just .gz. It took me awhile to realize it was a tarfile.)

Re: application dumping core during system reboot

Srimalik — Sun, 04 Nov 2007 23:49:39 GMT

>So the workaround may be simple, provide an >absolute path to vxsvc. (I assume that's >what you were running?)
Yes, I am running vxsvc.
Where do I need to give the absolute path? we are already using the absolute path in the rc script which starts vxsvc.

>(You're not in a directory that gets removed >out from under it?)

I think no, as the core is in / I think the working dir for a rc script is / by default.

(It would have been helpful if your attachment was .tar.gz or .tgz instead of just .gz. It took me awhile to realize it was a tarfile.)

Will keep that in mind for future.

Sri

Re: application dumping core during system reboot

Dennis Handly — Mon, 05 Nov 2007 02:21:46 GMT

>Where do I need to give the absolute path? we are already using the absolute path in the rc script which starts vxsvc.

Well, that's where I would expect it.
If not there, perhaps it forks and execs itself?

Re: application dumping core during system reboot

Srimalik — Mon, 26 Nov 2007 08:54:55 GMT

Hi Dennis,

> In the meantime, you should contact the
Response Center with your problem so they
can put more checking into dld.
Unfortunately if realpath doesn't work or
getcwd(3) fails, dld may give you an error
that it can't compute $ORIGIN and you'll
just abort with a nicer message.

Can't we fix this i.e cann't we prevent the abort instead of giving a nicer message? either in the application or the OS.

Thanks
Sri

Re: application dumping core during system reboot

Dennis Handly — Mon, 26 Nov 2007 11:18:07 GMT

>can't we prevent the abort instead of giving a nicer message? either in the application or the OS.

You can't fix something that hasn't been reported. Nor can you fix something if you can't duplicate. Currently we can only check for the symptoms of the lack of error detection in get_origin.

It might be interesting to see if exec(2) passed the absolute path to vxsvc. (Do you still have that corefile?) Or get the output of tusc to see what's different when it fails vs when it works.