Core dump while porting from 32 bit to 64 bit Itanium System

[Sandeep M]

Hi I am getting this core dump when i run an application built for Itanium system.
I tried to set break point in the main of the program..but it doesnt even reach there.All i can see is its crashing in strcpy method of dynamic loader ? Any thoughts on this ??

SEGV_ACCERR - Invalid Permissions for object
#0 0x9fffffffef7db8a0:0 in strcpy+0x20 () from /usr/lib/hpux64/dld.so
(gdb) bt
#0 0x9fffffffef7db8a0:0 in strcpy+0x20 () from /usr/lib/hpux64/dld.so
#1 0x9fffffffef7ad480:0 in _dlopen+0x860 () from /usr/lib/hpux64/dld.so
#2 0x9fffffffeec42f00:0 in __dlopen+0xe0 () from /usr/lib/hpux64/libdl.so.1
#3 0x9fffffffeec42920:0 in dlopen+0x40 () from /usr/lib/hpux64/libdl.so.1
#4 0x9fffffffef7ad480:0 in _dlopen+0x860 () from /usr/lib/hpux64/dld.so

[Dennis Handly]

Any reason you are porting to 64 bit mode? You don't have to do that to go to IPF.
Have you compiled with +w64bit?

Are you calling dlopen? Set a breakpoint there and see who is calling it. It seems that it is being passed a bad pointer. In gdb you can do:
(gdb) p /x $r32
(gdb) x /s $r32

If "break dlopen" doesn't work, you may have to use "catch load" first.

[Sandeep M]

Hi Dennis,
I'm trying to port from Tru64 to Hpux.I didnt specify +w64 bit flag .Let me try with this

Thanks,
Sandeep

[Sandeep M]

Hi Dennis,
I'm trying to port from Tru64 to Hpux.I didnt specify +w64 bit flag .Let me try with this

Thanks,
Sandeep

[Sandeep M]

Hi Dennis,
I could set break point on dlopen and then immediately after the first breakpoint being hit.i tried a single stepping which caused a core dump.

(gdb) break dlopen
Breakpoint 1 (deferred) at "dlopen" ("dlopen" was not found).
Breakpoint deferred until a shared library containing "dlopen" is loaded.
(gdb) r
Breakpoint 1, 0x9fffffffeec428f0:2 in dlopen+0x12 ()
from /usr/lib/hpux64/libdl.so.1
(gdb) s
Single stepping until exit from function dlopen,
which has no line number information.
Pid 22167 in trap loop, signal 11
warning: Temporarily disabling or deleting shared library breakpoints:
warning: Disabling breakpoint #1

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

[Sandeep M]

Program terminated with signal 11, Segmentation fault.
SEGV_ACCERR - Invalid Permissions for object
#0 0xc00000000006b8a0:0 in strcpy+0x20 () from /usr/lib/hpux64/dld.so
(gdb) bt
#0 0xc00000000006b8a0:0 in strcpy+0x20 () from /usr/lib/hpux64/dld.so
#1 0xc00000000003d480:0 in _dlopen+0x860 () from /usr/lib/hpux64/dld.so
#2 0xc000000000378f00:0 in __dlopen+0xe0 () from /usr/lib/hpux64/libdl.so.1
#3 0xc000000000378920:0 in dlopen+0x40 () from /usr/lib/hpux64/libdl.so.1
#4 0xc00000000003d480:0 in _dlopen+0x860 () from /usr/lib/hpux64/dld.so
(gdb) p /x $r32
$1 = 0x4c0
(gdb) x /x $r32
0x4c0: Cannot access memory at address 0x4c0
(gdb) x /s $r32
0x4c0:

[Dennis Handly]

>I'm trying to port from Tru64 to Hpux.

I thought Tru64 was already 64 bit and so you should be 64 bit clean? If not, you should be in 32 bit mode, unless you need the large address space.

What version of gdb do you have? This is not a complete stack trace so we can't figure out who is calling dlopen. Download the latest.

>(gdb) p /x $r32
$1 = 0x4c0
>(gdb) x /x $r32
0x4c0: Cannot access memory at address 0x4c0

This is not a valid pointer to a string, it seems truncated to 32 bits. You may have 64 bit porting issues.

[rick jones]

Just to be paranoid/certain, Sandeep - you are using the +DD64 compiler option to enable 64-bit compilation yes?

ISTR that while Alpha was "64-bit" there were some compiler tricks - not sure if they were enabled by default or not - which effectively took advantage of initial addresses being below a 32-bit boundary. So long as one didn't actually try to go beyond 32bits of memory one was OK. That being the case, I could see where there might be some lurking bugs in what someone would thing was "64-bit clean" Alpha code.

[Dennis Handly]

>Rick: you are using the +DD64 compiler option?

There is no need to ask since that is evident from the stack trace. Of course we don't know if this gcc, g++ or aC++ code.

[Sandeep M]

I'm using aCC compiler with
+DD64 option .All the libraries are ELF-64 bit code.

[Sandeep M]

The CFLAGS being used are
CFLAGS = +DD64 -D__ia64 -DHPUX -g +tru64 -g -AA

I later added +z option to generate pic code for shared libraries .But still the result is the same.The code crashes even before main of my program is reached.

[Dennis Handly]

>CFLAGS = +DD64 -D__ia64 -DHPUX -g +tru64
-AA

Remove -D__ia64 since that is the default.

>I later added +z option to generate pic code

The IPF default is PIC, +z is ignored.

You need to find who is calling dlopen. You should also install a newer gdb so your stack trace is complete. Also, when you get to dlopen, try tracing the stack there.

YOor previous output is probably all bogus. I wanted you to do this when you were in dlopen, not at the abort. So either use "frame 3" to get there or do this in dlopen:
(gdb) p /x $r32
(gdb) x /x $r32
(gdb) x /s $r32

You may also want to use "catch load" to see when shlibs are being loaded.

[Sandeep M]

Hi Dennis,
I set break points on dlopen and caught the library loads.It seems all the libraries got loaded.Just after this trace i did a single step.It crashed.

Breakpoint 1, 0x9fffffffeebf68f0:2 in dlopen+0x12 ()
from /usr/lib/hpux64/libdl.so.1
(gdb) bt
#0 0x9fffffffeebf68f0:2 in dlopen+0x12 () from /usr/lib/hpux64/libdl.so.1
#1 0x9fffffffef72f400:0 in real_load+0x900 () from /usr/lib/hpux64/libc.so.1
#2 0x9fffffffef72e9d0:0 in load_locale+0x180 () from /usr/lib/hpux64/libc.so.1
#3 0x9fffffffef72ba20:0 in find_locale+0xfa0 () from /usr/lib/hpux64/libc.so.1
#4 0x9fffffffef732620:0 in T_2d_2ca0_cl_do_setlocale+0xf0 ()
from /usr/lib/hpux64/libc.so.1
#5 0x9fffffffef727840:0 in setlocale+0x140 () from /usr/lib/hpux64/libc.so.1
#6 0x9fffffffeed35450:0 in __rw::__rw_use_c_lib_locale::__rw_use_c_lib_locale(char const*,int)+0x140 () from /usr/lib/hpux64/libstd_v2.so.1
#7 0x9fffffffeed383a0:0 in std::ctype::_C_initfacet(std::locale const&)
+0x60 () from /usr/lib/hpux64/libstd_v2.so.1
#8 0x9fffffffeed3d320:0 in std::locale::_C_install_facet(__rw::__rw_facet_base*,std::locale::id const&) const+0x120 () from /usr/lib/hpux64/libstd_v2.so.1
#9 0x9fffffffeed3c3c0:0 in std::locale::_C_make_facet(std::locale::id const&,bool,int,__rw::__rw_facet_base* (*)(int,char const*,unsigned long)) const
+0x3e0 () from /usr/lib/hpux64/libstd_v2.so.1
#10 0x9fffffffeee23050:0 in std::ios_base::Init::Init()+0x16c0 ()
from /usr/lib/hpux64/libstd_v2.so.1
#11 0x9fffffffef02e6e0:0 in __sinit_Usage_cc_+0x30 ()
at /opt/aCC/include_std/iostream:48
#12 0x9fffffffef79f050:0 in EM_mark_BOS+0x50 () from /usr/lib/hpux64/dld.so
(gdb) n
Single stepping until exit from function dlopen,
which has no line number information.

[Sandeep M]

Hi Dennis,
The compiler team asked me to install a patch for linker PHSS 34858.With the patch applied also it crashed.

this time its unable to load any library

0x9fffffffef7a5d00:1 in LE_rpath+0x1 () from /usr/lib/hpux64/dld.so
(gdb) n
Single stepping until exit from function LE_rpath,
which has no line number information.

Program received signal SIGBUS, Bus error
si_code: 1 - BUS_ADRALN - Invalid address alignment.
0x9fffffffef7a5d00:1 in LE_rpath+0x1 () from /usr/lib/hpux64/dld.so
(gdb) n
Single stepping until exit from function LE_rpath,
which has no line number information.

Program terminated with signal SIGBUS, Bus error.
The program no longer exists.

[Dennis Handly]

>I set break points on dlopen and caught the library loads. It seems all the libraries got loaded.Just after this trace i did a single step.It crashed.

I need to see the inputs into dlopen. Please print them before you continue. Also you neglected to mention you were were using aCC, since you mentioned CFLAGS.

You are currently initializating iostreams, are you threaded? If so, did you compile everything with -mt?

It appears you have massive corruption of of the runtime data structures. Do you have any static initializations?

Try "rbreak sinit" and see which of them are yours, before it aborts.

[Sandeep M]

Hi Dennis,
The mystery surrounding the core dump was attributed to a missing shared object file.
I was able to figure it out through gdb that it was this shared object /usr/lib/nls/loc/hpux64/locales.1/C which was being attempted to be loaded.
This was not present on the system.
Only C.iso88591 C.iso885915 C.utf8 were present.The only alternative was to copy/create a link to one of these shared object file.After making the copy , things started working fine.
I'm not sure if this shared object C was supposed to be installed during OS install and somehow went missing !

Thanks,
Sandeep

[Dennis Handly]

>I was able to figure it out through gdb that it was this shared object /usr/lib/nls/loc/hpux64/locales.1/C which was being attempted to be loaded. This was not present on the system.

This is not how dlopen is suppose to work. If it isn't there, dlopen returns an error and setlocale is suppose to recover from it. dlopen isn't suppose to abort.

>I'm not sure if this shared object C was supposed to be installed

No, the American Nerd (C) locale not suppose to have any libs in /usr/lib/nls/loc.

You may want to try a small test case that just calls setlocale(LC_ALL, "C") and see if you can duplicate the problem.

Did you ever print the inputs for dlopen and now setlocale? For setlocale, use $r33.

What version of libstd_v2.so.1 do you have?

[Sandeep M]

Hi Dennis,
Please let me know if this information is ok or do you need more info .
Thanks,
Sandeep

what ./usr/lib/hpux64/libstd_v2.so.1
./usr/lib/hpux64/libstd_v2.so.1:
HP aC++ for Integrity Servers B3910B A.06.10 [Mar 22 2006] C++ Standard Library (RogueWave Version 2.02.01)


SEGV_ACCERR - Invalid Permissions for object
bt
#0 0xc000000005035cb0:1 in strcpy+0x31 () from /usr/lib/hpux64/dld.so
(gdb) bt
#0 0xc000000005035cb0:1 in strcpy+0x31 () from /usr/lib/hpux64/dld.so
#1 0xc000000005047fa0:0 in _dlopen+0xac0 () from /usr/lib/hpux64/dld.so
#2 0xc000000005096020:0 in __dlopen+0xf0 () from /usr/lib/hpux64/libdl.so.1
#3 0xc000000005095a60:0 in dlopen+0x40 () from /usr/lib/hpux64/libdl.so.1
#4 0xc000000005047fa0:0 in _dlopen+0xac0 () from /usr/lib/hpux64/dld.so
(gdb) q
upc1>gdb -c core ng_nm_reader
HP gdb 5.4.0 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.4.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `ng_nm_reader'.
Program terminated with signal 11, Segmentation fault.
SEGV_ACCERR - Invalid Permissions for object
#0 0xc000000005035cb0:1 in strcpy+0x31 () from /usr/lib/hpux64/dld.so
(gdb) up
#1 0xc000000005047fa0:0 in _dlopen+0xac0 () from /usr/lib/hpux64/dld.so
(gdb) x /s $r33
0xa:

(gdb) up
#2 0xc000000005096020:0 in __dlopen+0xf0 () from /usr/lib/hpux64/libdl.so.1
(gdb) up
#3 0xc000000005095a60:0 in dlopen+0x40 () from /usr/lib/hpux64/libdl.so.1
(gdb) x /s $r33
0xa:

(gdb) up
#4 0xc000000005047fa0:0 in _dlopen+0xac0 () from /usr/lib/hpux64/dld.so
(gdb) x /s $r33
0x9fffffffffffcd10: "C"
(gdb) x /s $r32
0x9fffffffffffc4b1: "usr/lib/nls/loc/hpux64/locales.1/C"

[Dennis Handly]

>Please let me know if this information is ok or do you need more info .

Ok, all my previous questions may be little confusing, so from the start:

>HP aC++ for Integrity Servers B3910B A.06.10 [Mar 22 2006]

This is PHSS_34441, the latest. I don't know of any problems here.

SEGV_ACCERR - Invalid Permissions for object
#0 0xc000000005035cb0:1 in strcpy+0x31 /usr/lib/hpux64/dld.so

Ok, let's see what's strcpy is doing. Do:
(gdb) frame 0
(gdb) info reg
(gdb) disas 0xc000000005035cb0-0x30 0xc000000005035cb0+16*4

>(gdb) up

Hmm, frame 4 is bogus, since it matches frame 2. Perhaps getting a newer gdb may help?

Instead of doing this, use:
(gdb) frame 3
(gdb) info reg
(gdb) disas 0x9fffffffeec42920-0x40 0x9fffffffeec42920+16*4

Note you may want to set a breakpoint at dlopen to see how many times you hit it:

(gdb) b dlopen
(this returns a breakpoint number, say N)

(gdb) command N
p /x $r32
p /x $r33
x /s $r32
c
end

This will print the parms and continue.