Problem with BUS_ADRALN - Invalid address alignment on HP-UX B.11.22 U ia64 if linked with ptmalloc

[Centrify]

My progrom failed on hpux11.22 platform, to simplify my question, i wrote a very simple program to reproduce the question.

on hp 11.22, if the following source code linked with the famous multi-threads malloc lib "ptmalloc", it will coredump with the
error "BUS_ADRALN - Invalid address alignment".
This problem does not exist on hpux B.11.23 ia64.

I have been working on this for couple days and can not found out what is the root cause.

Can anyone give me any suggestion?

Thank you!


#include
#include
#include
#include
#include
#include

#include

#if !USE_MALLOC
#include
#else
#include "malloc-2.8.3.h"
#endif

/* Thread start function: display address near top of our stack,
and return upper-cased copy of argv_string */

static void *
thread_start(void *arg)
{
return 0;
}


int main(int argc, char *argv[])
{
pthread_t thread_id;
#if USE_MALLOC && USE_STARTER==2
ptmalloc_init();
printf("ptmalloc_init\n");
#endif
pthread_create(&thread_id, 0, &thread_start, 0);
sleep(60);
printf("return");
sleep(60);
return 0;
}


gcc -g -Wall -Wstrict-prototypes -D_REENTRANT -DUSE_TSD_DATA_HACK -DUSE_STARTE
R=2 -Isysdeps/pthread -Isysdeps/generic -I. -DUSE_MALLOC=1 -DTEST=1 t-test4.c -l
pthread libptmalloc3.a -o t-test4

-bash-3.00$ t-test4
ptmalloc_init
Bus error (core dumped)
-bash-3.00$ gdb t-test core
HP gdb 5.1.1 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.1.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..t-test: No such file or directory.

Reading symbols from t-test4...done.
Core was generated by `t-test4'.
Program terminated with signal 10, Bus error.
BUS_ADRALN - Invalid address alignment
#0 0x60000000c004fad0:0 in dld_bor_text_entry + 0x60 ()
from /usr/lib/hpux32/dld.so
(gdb) bt
#0 0x60000000c004fad0:0 in dld_bor_text_entry + 0x60 ()
from /usr/lib/hpux32/dld.so
#1 0x60000000c00ee0d0:0 in __spin_unlock_and_exit+0x230 ()
from /usr/lib/hpux32/libpthread.so.1
#2 0x60000000c00ee0d0:0 in __spin_unlock_and_exit+0x230 ()
from /usr/lib/hpux32/libpthread.so.1
(gdb)
(gdb) disas $pc-16*4 $pc+16*4
Dump of assembler code from 0x60000000c004fa90:0 to 0x60000000c004fb10:0:
0x60000000c004fa90:0 : nop.m 0x0
0x60000000c004fa90:1 : mov r3=b0
0x60000000c004fa90:2 : nop.b 0x0;;
0x60000000c004faa0:0 : st8 [r2]=r3,-8;;
0x60000000c004faa0:1 : mov r43=r16
0x60000000c004faa0:2 : nop.i 0x0;;
0x60000000c004fab0:0 : mov r44=r15;;
0x60000000c004fab0:1 : mov.m r3=ar.unat
0x60000000c004fab0:2 : nop.i 0x0;;
0x60000000c004fac0:0 : st8 [r2]=r3,-8;;
0x60000000c004fac0:1 : st8.spill [r2]=r8,-24
0x60000000c004fac0:2 : nop.i 0x0;;
0x60000000c004fad0:0 : stf.spill [r2]=f8,-16;;
0x60000000c004fad0:1 : stf.spill [r2]=f9,-16
0x60000000c004fad0:2 : nop.i 0x0;;
0x60000000c004fae0:0 : stf.spill [r2]=f10,-16;;
0x60000000c004fae0:1 : stf.spill [r2]=f11,-16
0x60000000c004fae0:2 : nop.i 0x0;;
0x60000000c004faf0:0 : stf.spill [r2]=f12,-16;;
0x60000000c004faf0:1 : stf.spill [r2]=f13,-16
0x60000000c004faf0:2 : nop.i 0x0;;
0x60000000c004fb00:0 : stf.spill [r2]=f14,-16;;
0x60000000c004fb00:1 : stf.spill [r2]=f15,-16
0x60000000c004fb00:2 : nop.i 0x0;;
0x60000000c004fb10:0 :
br.call.sptk.few b0=0x60000000c003bfb0

(gdb) info frame
Stack level 0, frame at 0x200000007ef6a178:
ip = 0x60000000c004fad0:0 in unwind_rp; saved ip 0x60000000c00ee0d0:0
(FRAMELESS), called by frame at 0x200000007ef6a178
Arglist at 0x200000007ef6a178, args:
Locals at 0x200000007ef6a178, Previous frame's sp is 0x200000007ef6a178
Saved registers:
gr32 at 0x200000007ef5a360, gr33 at 0x200000007ef5a368,
gr34 at 0x200000007ef5a370, gr35 at 0x200000007ef5a378,
gr36 at 0x200000007ef5a380, gr37 at 0x200000007ef5a388,
gr38 at 0x200000007ef5a390, gr39 at 0x200000007ef5a398,
gr40 at 0x200000007ef5a3a0, gr41 at 0x200000007ef5a3a8,
gr42 at 0x200000007ef5a3b0, gr43 at 0x200000007ef5a3b8,
gr44 at 0x200000007ef5a3c0, gr45 at 0x200000007ef5a3c8,
gr46 at 0x200000007ef5a3d0, gr47 at 0x200000007ef5a3d8,
gr48 at 0x200000007ef5a3e0, gr49 at 0x200000007ef5a3e8,
gr50 at 0x200000007ef5a3f0, gr51 at 0x200000007ef5a400,
gr52 at 0x200000007ef5a408, gr53 at 0x200000007ef5a410
(gdb)
(gdb) info reg r2
gr2: 200000007ef69f78
(gdb)

[Dennis Handly]

11.22 is no longer supported, why are you even using it?

The problem is related to the fact that ALL stack frames must be 16 byte aligned. In your case, you have an address 0x200000007ef69f78 that is only 8 byte aligned.
R12, SP should also be misaligned in your case.

Does your ptmalloc know that all results must be 16 byte aligned on Integrity?

Linking your application with -Wl,-Bimmediate may make it go away since that removes BOR.

[Centrify]

Dennis Handly.
Thank you very much, you are so kindly and your response is so quick.
Now, i know what is the root cause.

-->11.22 is no longer supported, why are you even using it?
yes, our software still need to support it.

-->Does your ptmalloc know that all results must be 16 byte aligned on Integrity?
my ptmalloc just return 8bytes aligned address, do you mean the function malloc
on hpunx B.11.22 always return 16bytes aligned address? otherwise, when pthread_create
calls malloc, how does malloc know to return a 16bytes aligned address or not.
and is it not the same case on hpux B.11.23 ia64 that malloc will return 16bytes aligned
address?

-->Linking your application with -Wl,-Bimmediate may make it go away since that removes BOR

link with -Wl,-B,immediate can just fix the problem in the above case, if i change the above
code main function to the following, still have the same problem.
int main(int argc, char *argv[])

{
pthread_t thread_id[20];
int num;
#if USE_MALLOC && USE_STARTER==2
ptmalloc_init();
printf("ptmalloc_init\n");
#endif
for(num=0; num<20; num++)
pthread_create(&thread_id[num], 0, &thread_start, 0);
for(num=0; num<20; num++)
pthread_join(thread_id[num], NULL);
sleep(10);
printf("return\n");
sleep(10);
return 0;
}


-bash-3.00$ make posix-explicit
make THR_FLAGS='-D_REENTRANT -DUSE_TSD_DATA_HACK -DUSE_STARTER=2' \
THR_LIBS=-lpthread \
OPT_FLAGS='-g -Wl,-B,immediate -O2 ' SYS_FLAGS='' CC='gcc' \
INC_FLAGS='-Isysdeps/pthread -Isysdeps/generic -I.' \
M_FLAGS='-DTHREAD_STATS=1 '
make[1]: Entering directory `/home/michael/CDC/build/GARFIELD-M1/external/build/
ptmalloc3'
gcc -c -g -Wl,-B,immediate -O2 -Wall -Wstrict-prototypes -D_REENTRANT -DUSE_TS
D_DATA_HACK -DUSE_STARTER=2 -Isysdeps/pthread -Isysdeps/generic -I. -DTHREAD_STA
TS=1 -DMSPACES=1 ptmalloc3.c
ptmalloc3.c:511: warning: 'save_memalign_hook' defined but not used
gcc: -B: linker input file unused because linking not done
gcc: immediate: linker input file unused because linking not done
gcc -c -g -Wl,-B,immediate -O2 -Wall -Wstrict-prototypes -D_REENTRANT -DUSE_TS
D_DATA_HACK -DUSE_STARTER=2 -Isysdeps/pthread -Isysdeps/generic -I. -DTHREAD_STA
TS=1 -DONLY_MSPACES -DUSE_LOCKS=0 -DPROCEED_ON_ERROR malloc.c
malloc.c: In function `usage_error':
malloc.c:3628: warning: unused variable `err_mem'
malloc.c:3629: warning: unused variable `err_m'
gcc: -B: linker input file unused because linking not done
gcc: immediate: linker input file unused because linking not done
ar cr libptmalloc3.a ptmalloc3.o malloc.o
ranlib libptmalloc3.a

gcc -g -Wl,-B,immediate -O2 -Wall -Wstrict-prototypes -D_REENTRANT -DUSE_TSD_D
ATA_HACK -DUSE_STARTER=2 -Isysdeps/pthread -Isysdeps/generic -I. -DUSE_MALLOC=1
-DTEST=1 t-test4.c libptmalloc3.a -lpthread -o t-test4

-bash-3.00$ t-test4
ptmalloc_init
Bus error (core dumped)
-bash-3.00$ gdb t-test4 core
HP gdb 5.1.1 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 5.1.1 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `t-test4'.
Program terminated with signal 10, Bus error.
BUS_ADRALN - Invalid address alignment
#0 0x60000000c014f140:0 in __restore_context+0x260 ()
from /usr/lib/hpux32/libpthread.so.1
(gdb) bt
#0 0x60000000c014f140:0 in __restore_context+0x260 ()
from /usr/lib/hpux32/libpthread.so.1
warning: Attempting to unwind past bad PC 0x60000000c014f140
#1 0x60000000c014f5e0:0 in __thread_start0+0 ()
from /usr/lib/hpux32/libpthread.so.1
(gdb) bt
#0 0x60000000c014f140:0 in __restore_context+0x260 ()
from /usr/lib/hpux32/libpthread.so.1
warning: Attempting to unwind past bad PC 0x60000000c014f140
#1 0x60000000c014f5e0:0 in __thread_start0+0 ()
from /usr/lib/hpux32/libpthread.so.1
(gdb) disas $pc-16*4 $pc+16*4
Dump of assembler code from 0x60000000c014f100:0 to 0x60000000c014f180:0:
0x60000000c014f100:0 <__restore_context+0x220>:
ld8.fill r26=[r33],16
0x60000000c014f100:1 <__restore_context+0x221>:
ld8.fill r27=[r34],16
0x60000000c014f100:2 <__restore_context+0x222>: nop.i 0x0;;
0x60000000c014f110:0 <__restore_context+0x230>:
ld8.fill r28=[r33],16
0x60000000c014f110:1 <__restore_context+0x231>:
ld8.fill r29=[r34],16
0x60000000c014f110:2 <__restore_context+0x232>: nop.i 0x0;;
0x60000000c014f120:0 <__restore_context+0x240>:
ld8.fill r30=[r33],16
0x60000000c014f120:1 <__restore_context+0x241>:
ld8.fill r31=[r34],24
0x60000000c014f120:2 <__restore_context+0x242>: nop.i 0x0;;
0x60000000c014f130:0 <__restore_context+0x250>:
mov.m ar.unat=r36
0x60000000c014f130:1 <__restore_context+0x251>: nop.m 0x0
0x60000000c014f130:2 <__restore_context+0x252>: nop.i 0x0;;
0x60000000c014f140:0 <__restore_context+0x260>:
ldf.fill f2=[r33],32
0x60000000c014f140:1 <__restore_context+0x261>:
---Type to continue, or q to quit---
ldf.fill f3=[r34],32
0x60000000c014f140:2 <__restore_context+0x262>: nop.i 0x0;;
0x60000000c014f150:0 <__restore_context+0x270>:
ldf.fill f4=[r33],32
0x60000000c014f150:1 <__restore_context+0x271>:
ldf.fill f5=[r34],32
0x60000000c014f150:2 <__restore_context+0x272>: nop.i 0x0;;
0x60000000c014f160:0 <__restore_context+0x280>:
ldf.fill f6=[r33],32
0x60000000c014f160:1 <__restore_context+0x281>:
ldf.fill f7=[r34],32
0x60000000c014f160:2 <__restore_context+0x282>: nop.i 0x0;;
0x60000000c014f170:0 <__restore_context+0x290>:
ldf.fill f8=[r33],32
0x60000000c014f170:1 <__restore_context+0x291>:
ldf.fill f9=[r34],32
0x60000000c014f170:2 <__restore_context+0x292>: nop.i 0x0;;
0x60000000c014f180:0 <__restore_context+0x2a0>:
ldf.fill f10=[r33],32
End of assembler dump.
(gdb) info reg r33
gr33: 200000007ef6c4b8
(gdb) info frame
Stack level 0, frame at 0x200000007ee34ff0:
ip = 0x60000000c014f140:0 in __restore_context; saved ip 0x60000000c014f5e0:0
(FRAMELESS), called by frame at 0x200000007ee34ff0
Arglist at 0x200000007ee34ff0, args:
Locals at 0x200000007ee34ff0, Previous frame's sp is 0x200000007ee34ff0
Saved registers:
gr32 at 0x200000007edf5000, gr33 at 0x200000007edf5008,
gr34 at 0x200000007edf5010, gr35 at 0x200000007edf5018,
gr36 at 0x200000007edf5020, gr37 at 0x200000007edf5028,
gr38 at 0x200000007edf5030, gr39 at 0x200000007edf5038


Is there anything wrong with the above gcc compile command?
If no, is there any other way to fix this problem?
Or the only solution is to change the ptmalloc to return
16bytes aligned addess. because the ptmalloc is a free
software and it's code is opotimized for the case of 8bytes
aligned, i tried changing the code to return 16bytes aligned
address and failed. and at the same time i am not sure if the
performance will be reduced if 16bytes aligned address is returned.

Btw, what is the meaning of BOR, you has memtioned in other thread it is
Bind On Reference, I still do not understand, and what is the role of BOR.

Thank you.

[Dennis Handly · Accepted solution]

>our software still need to support it.

HP no longer supports 11.22.

>my ptmalloc just return 8 bytes aligned address

Then this is not a valid malloc replacement for Integrity.

>do you mean the function malloc on HP-UX B.11.22 always return 16 bytes aligned address?

On all HP-UX IPF systems, the frames and heap regions must be 16 byte aligned, to support long double and floating point register fills/spills.

>when pthread_create calls malloc, how does malloc know to return a 16 bytes aligned address or not.

malloc always returns at least 16 byte aligned.

>B.11.23 IPF that malloc will return 16 bytes aligned address?

Yes, this is part of the IA64 runtime architecture.

>if I change the above code main function to the following, still have the same problem.

The problem just moved elsewhere.

>Is there anything wrong with the above gcc compile command? ... Or the only solution is to change the ptmalloc to return 16 bytes aligned addess.

You must fix ptmalloc function or remove it and use the libc malloc.

>because the ptmalloc is a free software and it's code is optimized for the case of 8 bytes aligned

This software is broken if it doesn't allow you to adjust your alignment based on hardware constraints.

>I am not sure if the performance will be reduced if 16 bytes aligned address is returned.

Well, it may waste space. And why do you care since the OS isn't supported?

>what is the meaning of BOR, you mentioned in other thread it is Bind On Reference

BOR is when dld delays the binding on function calls until the call, not at startup. This may help the application start quicker.

[Centrify]

Dennis Handly.
Thank you very much, your response is so quick.
I just read the document named <>, As you said, the stack pointer must always be aligned at a 16-byte
boundary.

And now i am working to change my ptmalloc to return a 16-bytes aligned address.

And there is another question i can not understand, why my software will not break on HPUX B.11.23 ia64? Our software is compiled as a ELF-32 executable object on both 11.22 ia64 and 11.23 ia64.

-bash-3.00$ file t-test1
t-test1: ELF-32 executable object file - IA64
-bash-3.00$

What is the differenc between 11.22 ia64 and 11.23 ia64?

[Dennis Handly]

>why my software will not break on HP-UX B.11.23 ia64? What is the difference between 11.22 ia64 and 11.23 ia64?

There are hundreds of differences, the major one is that 11.22 isn't supported and 11.23 has lots and lots of new features/patches.

In particular, as I was looking at your issue, I asked that same question and the only thing that made sense is that libpthread now calls mmap(2) on 11.23 to get the thread stack.

[Centrify]

Dennis.
Thanks for your and HP's excellent support.

[Centrify]

Thanks. I have gotten the solution.