PA Risk Executable on Itanium Server

[Rao Uppuluri]

Hi
I have an executable compiled on HP-UX11i(11.11) that I ran on an HP-UX11i(11.23) Itanium server and it crashed. Gave following error:
[HP ARIES64]: Core file for 64 bit PA-RISC application
[HP ARIES64]: saved to core..
Memory fault(coredump)

BUT, When I ran the same exec on the Itanium server by using wdb (debuuger) it runs successfully! Any ideas please?

Thank you
Rao

[Dennis Handly · Accepted solution]

Well, if it aborts, you should just debug from the core file. Getting a stack trace would be the first step.

I've had a few cases where it runs in the debugger. The obvious cases are uninitialized variables.
I've seen other cases where the debugger's private mapping of shlibs causes readonly strings to be writable and then it won't abort. But that should fail on PA too.

[Rao Uppuluri]

Dennis
Tks for the reply. I typed following (I am not familiar yet with debugging core files)

wdb -c core.usermaster usermaster

(usermaster is exec name)
It produced following output.

Detected 64-bit executable.
Invoking /usr/ccs/bin/gdb64.
HP gdb 3.4 for PA-RISC 2.0 (wide), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 3.4 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `usermaster'.
Program terminated with signal 11, Segmentation fault.
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
1029 multibyte.c: No such file or directory.

Does this help you find anything? Also, is it possible for you to give me instructions to debug a core file, please? (to get stack trace)

[Srimalik]

you have to give "bt" command to get a stack trace.

fnc_mbStrLen is causing a segfault.
Please confirm that 0x80000001000065d7 is valid address. You can try printing the string if it is a invalid address the stack will change(print function will dump somewhere). Then you need to trace this variable.

You can use this link for more gdb commands

http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801?ciid=e8080f1bace021100f1bace02110275d6e10RCRD

[Rao Uppuluri]

Hi Srikrishnan
I did the following gdb core on the Itanium server

and typed bt to get stack trace.
========= gdb output==========
(gdb) bt
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
#1 0xc0000000080693b0 in fnc_StrLen (string=0x9fffffffe00033a8 "\n")
at str.c:767
#2 0xc00000000805ef44 in fnc_LoadIniFile (
IniFilename=0x9fffffffdfef87b0 "app.ini") at param.c:362
#3 0xc00000000805f880 in fnc_GetParamStr (
Section=0xc00000000801ca40 "Memory",
Item=0xc00000000801ca48 "TakeOverSIGUSR1", Default=0x0) at param.c:694
#4 0xc00000000805fad8 in fnc_GetParamLong (
Section=0xc00000000801ca40 "Memory",
Item=0xc00000000801ca48 "TakeOverSIGUSR1", Default=0) at param.c:754
#5 0xc00000000804f63c in EachTime () at mem.c:692
#6 0xc00000000804dd20 in fnc_Malloc (size=1024) at mem.c:214
#7 0xc00000000805ee98 in fnc_LoadIniFile (
IniFilename=0x9fffffffdfef87b0 "app.ini") at param.c:342
#8 0xc00000000805f880 in fnc_GetParamStr (Section=0xc000000008018938 "Debug",
Item=0xc000000008018ad8 "Dgn", Default=0xc000000008018ae0 "Off")
at param.c:694
#9 0xc00000000804169c in fnc_InitDgn () at dgn.c:411
#10 0xc000000007fc3300 in cdm_Initialise () at cdmslab.c:7543
#11 0xc000000007e49fbc in link_Initialise () at appclk.c:257
#12 0x4000000000002cb4 in main () at usermaster.c:59

(gdb) p 0x80000001000065d7
$1 = 9223372041149769175
(gdb)
=====+++++ end gdb output =====
Does that mean it is a valid address?

Tks

[Dennis Handly]

(gdb) p 0x80000001000065d7
$1 = 9223372041149769175
>Does that mean it is a valid address?

No, you just used calculator mode to convert hex to decimal.

You need to do:
(gdb) p string
(gdb) x /8gx 0x80000001000065d7-7

The first uses debug info. The second debugs at the machine code level.

For debugging core files there is more documentation here:
http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=09080f1bace021100f1bace02110275d6e10RCRD
See: Debugging core files using HP WDB

[Steven E. Protter]

Shalom Rao,

For this and other reasons, you want to recompile.

The emulation that permits PA-RISC an executable to run on Intanium servers comes with a 70% performance penalty.

SEP

[Dennis Handly]

>SEP: For this and other reasons, you want to recompile.

Because of the advanced compiler debugging options? ;-)
There is +check= (runtime checking) and +wlint.

>The emulation that permits PA-RISC an executable to run on Integrity servers comes with a 70% performance penalty.

It highly depends on what PA box you were using and whether the application was CPU bound. See:
http://www.hp.com/go/aries

[James R. Ferguson]

Hi:

The degree of performance penality under Aries is dependent upon both options configured and work type. These are summarized in the ARIES manpages:

http://docs.hp.com/en/B3921-60631/ARIES.5.html

Regards!

...JRF...

[Dennis Handly]

>Srikrishan: Please confirm that 0x80000001000065d7 is valid address.

This may have nothing to do with the abort. I.e. it's just a parm of the aborting function.

The real problem is on line 1029 of multibyte.c. What's on that line?
Since the executable has debug info, why not use it? ;-)

[Rao Uppuluri]

Thank you all for quick responses.

* I can't really re-compile on Itanium because I am using a third party library calls and they are not supported under Itanium. So, I really want to compile it under PA Risk and run it on Itanium. If it helps, I ran this same PA executable on a different Itanium server (located at a diff location in the company) and it works! That server is also has 11iv2. Couldn't really figured out the differences between our servers yet.

Dennis,
>The real problem is on line 1029 of multibyte.c. What's on that line?
>Since the executable has debug info, why not use it? ;-)

Can you please elobarate on how I can examine this issue?
Thank you

[Dennis Handly]

>I ran this same PA executable on a different Integrity server and it works! That server is also has 11iv2. Couldn't really figured out the differences between our servers yet.

Check the patch levels of Aries, libc and linker.

>>The real problem is on line 1029 of multibyte.c. What's on that line?

>Can you please elaborate on how I can examine this issue?

I assume you have the source file multibyte.c?
Otherwise you could do "info locals".

Or you can debug at machine level:
(gdb) disas $pc-4*20 $pc+4*4
(gdb) info reg

[Rao Uppuluri]

Dennis
I am getting the patch info for linker,aries and libc.
I don't have source for multibyte.c ( there is a manpage for multibyte).
when I tried the steps you provided in gdb, it provided output that I can't comprehend.

TFYH
Rao

[Dennis Handly]

>I don't have source for multibyte.c (there is a manpage for multibyte).

The man page for multibyte(3C) is HP's code. fnc_mbStrLen is your code.

>when I tried the steps you provided in gdb, it provided output that I can't comprehend.

;-)
Just attach them to the thread and I'll look at them.

[Don Morris_1]

Not to disagree slightly with Dennis as to the pertinence of the address -- but with that address being just over 4Gb, and maxdsiz_64bit defaulting to 4Gb... I can't help but wonder if you did something like realloc() of an array without checking the return for failure and assuming the reallocation succeeded... then walking off the end of the array into unallocated virtual memory.

Based on perusing documentation -- "maint info sections" should dump the executable+core memory ranges... it might be worth seeing what's there (and what isn't). If the data ends at 0x10000000 (or very close... you fail the allocation that pushes you over, it doesn't mean you always get exactly there).. that would fit.

[Srimalik]

Rao,

The top two frames in the stack are:
#########################################
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
#1 0xc0000000080693b0 in fnc_StrLen (string=0x9fffffffe00033a8 "\n")
at str.c:767
##########################################
In frame 1 (fnc_StrLen) it seems you are trying to calculate the length of string "\n"
which should be 1.

Then from this function you are calling fnc_mbStrLen with a empty string(length 0).
Maybe you should look into this.

Secondly the address passed to both these functions are in different areas(not near each other 0x8000000 and 0x9fffffff).

How does "\n" becomes ""...you need to trace the flow in your code.

Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
Note that the parameters are changed.

It may help.

-Sri

[Don Morris_1]

0x9ffffff.... would be at the top of the private octant for data/heap/mmf/stack, so is very likely a stack variable address. 0x80000... is more likely a heap allocation address [or is meant to be but may rather be a bad pointer arithmetic result after a failed allocation, hence my query about a realloc() in this path]. That's because it is near the low end of the octant (where the heap/BSS starts and then grows towards the memory mappings, thread stacks and main process stack that live on the other end and come towards lower addresses).

[Srimalik]

Hi, Don
I am not familiar with the octant concept.
As per your comments
0x80000... should heap area
0x9ffff... should be on stack

I see a different range of address in frame 8
i.e 0xc0000000080.. It lies near to the code seg, are these the environment variables?

It may be off topic for this thread, may not be related to the thread in any way.

[Dennis Handly]

>Don: Not to disagree slightly with Dennis as to the pertinence of the address -- but with that address being just over 4Gb, and maxdsiz_64bit defaulting to 4Gb.

That address is trivially close to the start of the PA64 data area, 26071 bytes.

I'm assuming two things:
1) If 64 bit, it needs more than 4 Gb. ;-)
2) The debugger would give an error when tracing the stack if the "string" had a bad address, rather than just "".
#0 fnc_mbStrLen (
string=0x80000001000065d7 )

This address is big for IPF64 but not for PA64.
Also, the debugger won't complain about bad alignment though.

>Don: Based on perusing documentation - "maint info sections" should dump the executable+core memory ranges.

A better option would just be to use gdb's "info file".

Or better yet, use the command:
$ elfdump -S -o core

>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.

Yes, looking at the source would help.

>Don: 0x80000... is more likely a heap allocation address

Or a global/static variable. Try using:
info sym 0x80000001000065d7

>Sri: I am not familiar with the octant concept.

(PA has quadrants, IPF has octants. They are related to space or region registers.)

>0x80000... should heap area
>0x9ffff... should be on stack
>I see a different range of address in frame 8. i.e 0xc0000000080.. It lies near to the code seg, are these the environment variables?

No, these are shlib text, or shared memory.

[Rao Uppuluri]

Hello all,

My maxdsiz_64bit is 4GB.

I don't have source multibyte.c, may be it is part of the 3rd party libraries I am using.
I am attaching the gdb and elfdump output.

Thank you

[Srimalik]

>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.


Did you try this thing ?

[Rao Uppuluri]

Sri
>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.

I can't debug at that level (I don't know how, I am calling a 3rd party library and program crashes)

[Srimalik]

I thought its your code as the stack shows many fnc_**** names.

Just to confirm, is fnc_InitDgn your code or its also in some third party library? which third party library?

May be I am confused by the naming convention used.

From where did you get the third party libraries with debugging symbols?

PS: Every answer does not deserve 10 points

http://forums11.itrc.hp.com/service/forums/helptips.do?#28

[Rao Uppuluri]

No, fnc_InitDgn is not in my code. I am guessing it is coming from 3rd party(An accounting software function calls) or from HP library.

I compiled the code on PA with -g option and debugged it on Itanium using wdb and from that session I am getting all the info you see here.

In any case, looks like no one really surprised by the fact that the exec runs fine through debugger but crashes when run by itself! ( I have seen similar behaviour with some VB code but never understood why)

Tks
Rao

[Srimalik]

Hi, Rao
Its very strange that the 3rd party is giving you libs with full debugging symbols. Not many vendors do this(never seen this for any HP library). Are you intentionally using debug version of the third party libs?


anyway, If you know who own this lib and have a support contract; you can involve them to look into this. You can show them the stack and provide them the core file.

as a call to fnc_mbStrLen("") from fnc_StrLen("\n") seems fishy to me \, I am wondering why would somebody do that?

-Sri