shared libraries and $ORIGIN

[howard rutiezer_1]

RE: shared libraries

We have software that runs on different vendors platforms. For HP-UX we want to use shared libraries with each library in a different directory.

Consider the following example which are programs extracted and modified from the HP-UX Linker and Libraries and User.s Guide. There are three programs, testlibfoo3.c, libfoo.c and libfoo2.c. The sequence in running the programs is:


1) testlibfoo3 in directory mydirA loads shared library libfoo.sl in mydirB and executes code
in libfoo.sl
2) libfoo loads shared library libfoo2 in mydirB, executes code in libfoo2, then unloads libfoo2.
3) testlibfoo3 unloads shared library libfoo.sl

Below is a description of what we did, but it may be easier to say how to do it correctly instead of finding error(s) in the way we tried to do this.

We think part of the code to do this can use the $ORIGIN option from the linker, ld.
The code to reproduce the problem we see is in the attachment of appendices to this note. The code for testlibfoo3, libfoo.c and libfoo2.c is in Appendices I, II and III, respectively. The code to compile, link and run
each of them is contained as comments in each of the appendices.

If the three files are in the same directory are compiled and linked then testlibfoo3 runs correctly.
I do this first to be sure the files are compiled and linked correctly.

The man page for ld says more information on the $ORIGIN option is available in the HP-UX Linker and Libraries User.s Guide. I looked in the HP-UX Linker and Libraries User.s Guide, document B2355-90655, November 1997, and do not understand how to use $ORIGIN.

I did see the man page for ld has a typo in the example:
$ ld -dynamic main.o -L /usr/lib/ -lc +B $ORIGIN
in which "+B" should be "+b"

My testing was done in the tcsh shell with these environment variables:

[qa00hpu:/home/hrutieze/bugs/dlopen/ex7/mydirA]536 % env | egrep 'ORIGIN|SHLIB_PATH'
SHLIB_PATH=./
ORIGIN=./

This what I saw from in running a.out

[qa00hpu:/home/hrutieze/bugs/dlopen/ex8/mydirA]822 % ./a.out
- Version 0.4
- testlibfoo3: Use shl_load for shared library libfoo.sl
- testlibfoo3: Error loading libfoo.sl: No such file or directory

This same information is contained in the attachment.

Thanks,
Howard

[ranganath ramachandra]

this is how the $ORIGIN thing works:

you put "$ORIGIN" (literally) in the embedded path (through +b, not sure if it works for path recorded through +cdp) at link time. at load-time (run-time), the loader will substitute "$ORIGIN" with the full path of the directory where the current load module (executable or shared library) was loaded from.

suppose you used a link line like

ld -b x.o -o libx.sl +b \$ORIGIN -ly

if, in a process, libx.sl was loaded from /usr/local/lib, then the loader will look for liby.sl in /usr/local/lib.

a typical example: suppose you ship application programs and binaries such that the programs reside in /bin and libraries are in /lib, where basedir is a base directory that the installing user can choose. you could build an application with a link line like

ld *.o -o app -lx ... +b \$ORIGIN/../lib

so that regardless of which base directory your application program and binaries are installed in, libx.sl (etc) can be picked from the neighbouring 'lib' directory.

make sure you follow proper shell/makefile syntax (wrt escape characters etc) to get the string "$ORIGIN" (without the quotes, of course) into the embedded path.

[howard rutiezer_1]

I included an attachment as a text file that
got mangled. The text file contains my description and a reproducer, containing the three files testlibfoo3.c, libfoo.c, libfoo2.c, as Appendices I, II and III. Will try the attachment again as a Microsoft Word file.

The command used in testing is
cc -g -O0 -Wl,+b \$ORIGIN:\$ORIGIN../mydirB testlibfoo3.c

chatr of a.out shows:
a.out:
shared executable
shared library dynamic path search:
SHLIB_PATH enabled first
embedded path enabled second $ORIGIN:$ORIGIN../mydirB
shared library list:
dynamic /usr/lib/libc.2
[ ... deleted lines ... ]


and environments are set this way and a.out is in directory mydirA
% env | egrep â SHLIB_PATH|ORIGINâ
SHLIB_PATH=./
ORIGIN

[howard rutiezer_1]

Bad character again at end of post
Dec 30, 2004 14:28:53 GMT
although it did not look that way to me when
I did it.


% env | egrep 'SHLIB_PATH|ORIGIN'
SHLIB_PATH=./
ORIGIN=./

Is there is a reference to HP documentation that completely explains how to use $ORIGIN?

I found incomplete documentation in the man page for ld and in the HP-UX Linker and Libraries User's Guide, HP 9000 Computers, document B2355-90655, November 1997, as described in the attachment with the prior posting.

[howard rutiezer_1]

I've gotten a little bit further.
If in the shl_load in testlibfoo3 one adds the flag DYNAMIC_PATH to this statement
if ((hndl_foo1 =
shl_load("libfoo.sl",
BIND_IMMEDIATE |BIND_VERBOSE|DYNAMIC_PATH,0)) == NULL)

then the program gets a bit further, it
loads libfoo.sl.

Let me look a little further into this after the holiday weekend. I think libfoo which is supposed to load libfoo2 does not use DYNAMIC_PATH and I will change that when I try again.

Let's put this on hold for now.

Thanks,
Howard

[howard rutiezer_1]

Continuing on with tests ...

See the attachemt again --

- Did NOT get $ORIGIN with embedded path to work.
- Did get SHLIB_PATH to work.


Testing was done with SHLIB_PATH set in each of these two ways:

When I used this SHLIB_PATH where "./" is "mydirA"

SHLIB_PATH=./:./../mydirB

the test ran successfully.
I think because dld.sl found the libraries to be loaded along the search path
specified by SHLIB_PATH.


When I changed this same environment variable to
SHLIB_PATH=./
the test hung trying to shl_load "libfoo.sl".

When I use a non-existent directory like
SHLIB_PATH=./:./../mydirX
the tests hang as in the previous case.


If you look at appendix I, you can see libtestfoo3.c was compiled with

cc -g -O0 -Wl,+b \$ORIGIN:\$ORIGIN../mydirB testlibfoo3.c

and that "chatr a.out" show the embedded path:

a.out:
shared executable
shared library dynamic path search:
SHLIB_PATH enabled first
embedded path enabled second $ORIGIN:$ORIGIN../mydirB
shared library list:
dynamic /usr/lib/libc.2

[ ... deleted lines ... ]


The directory structure for the testing is


|
mydirA | mydirB
+---------+-------+
| |
| |
a.out libfoo.sl, libfoo2.sl
created created from libfoo.c and
from libfoo2.c, respectively
testlibfoo3


The C code is

if ((hndl_foo1 = shl_load("libfoo.sl",
BIND_IMMEDIATE|BIND_VERBOSE|DYNAMIC_PATH,0)) == NULL)


This was tested with:

cpp.ansi: HP92453-01 B.11.11.06 HP C Preprocessor (ANSI)
ccom: HP92453-01 B.11.11.06 HP C Compiler
/usr/ccs/bin/ld: 92453-07 linker linker ld B.11.30 011005


Am I compiling or linking the program incorrectly?

As Ranganath says, "at load-time (run-time), the loader will substitute '$ORIGIN' with the full path of the directory where the current load module (executable or shared library) was loaded from." for both occurances of $ORIGIN ?

If so, why does this fail given the dirctory
structure shown above? And why hang instead
of returning with an error?

A different problem?

[Ermin Borovac]

I might be on the wrong track but it seems to me that you are missing additional slash character when linking.

You seem to currently have

+b \$ORIGIN../mydirB

Try linking with

+b \$ORIGIN/../mydirB

[ranganath ramachandra]

i'm sorry for telling you so late: for explicitly loaded shared libraries, the $ORIGIN stuff works only if you use dlopen and not if you use shl_load. do you have a reason not to use (the more portable) dl* api instead ?

try this:
----
cat > main.c << \!
#include
main () {
void (*foo)();
void *dl = dlopen("liblib.sl", RTLD_LAZY);
if (!dl) {
puts(dlerror());
exit(1);
}
foo=dlsym(dl, "foo");
if (foo)
foo();
dlclose(dl);
exit(0);
}
!
cat > lib.c << \!
foo () {
puts("yo");
}
!
cc main.c -o main -ldld -Wl,+b,\$ORIGIN/../lib
cc -b lib.c -o liblib.sl
# mkdir bin
mv -f main bin
# mkdir lib
mv -f liblib.sl lib
bin/main
----

this works for me, at least with the latest linker/loader patch (PHSS_30967 for 11.00).

[howard rutiezer_1]

Ermin,

Yes, as you said, a missing slash during
linking.

Ranganath,

I tried what you said and it worked as you
described.

This is what I think happens:
The first program, "main", loads "foo" and
calls the module "foo" in it.

Additional question:
How can the second loaded program, the shared library "foo", load a shared library "foo2" located in another directory, say "lib2" and call a module, say, "infoo2" in shared library foo2 using the $ORIGIN mechanism you showed in your post of 6-Jan?



And then go on this way, each shared library, possibly in a different directory calling into a shared library that may be in another directory. At run-time the relative path locations of the shared libraries are known, relative to the first program run,
"bin/main", in your example.

This software system of which this code is a part, runs on different vendors products.
I was told the reason "shl_load" is used is
historical, that is, it is used because "shl_load" was available at the
time and the dl* APIs were not.

I saw one paragraph in the HP-UX Linker and Libraries User's Guide manual, but nothing on use of more than one occurance of $ORIGIN
in a pathlist. Something like your example:
cc main.c -o main -ldld \
-Wl,+b,\$ORIGIN/../lib:\$ORIGIN/../lib2
It seems to work, but will the loader replace
each occurance of $ORIGIN with full path name
of the location of the executable or shared
library?

Can you point me to documentation that describes use of $ORIGIN combined with the dl* APIs? I was initially told "shl_load" and $ORIGIN work together because someone
said they saw it in HP documentation -- I
don't know where -- and it worked on a
version of Linux.

Thanks,
Howard

[ranganath ramachandra]

the missing slash does not seem to be a problem, the loader is putting in a /-terminated string in place of $ORIGIN.

there is very little documentation yet which i can point you to. i will make sure that something is done about it.

there is no limit to the number of occurences of "$ORIGIN", you can use as many as you want.

the loader will replace all instances of $ORIGIN in all embedded paths (executable and shared libraries). consider this case:
a.out->libx.sl->liby.sl->libz.sl
when libz.sl is loaded because of liby.sl, the search path will include all the embedded paths of a.out, libx.sl and liby.sl.

so you could have a.out having "$ORIGIN/../lib" in its embedded path and shared libraries having "$ORIGIN" in theirs.

as far as i know, no other platform has the shl_* api's, so i think it could not have been working in linux.

[howard rutiezer_1]

I was wrong about shl_load and Linux.

I tried the example you gave in the post of 6-Jan-2005 and it works exactly as you said.

I changed my example to use the dl* APIs and
still have a problem. The shared library, libfoo.sl, loading another shared library, libfoo2, to call a routine in it, fails.

Could you extend your example of 6-Jan do this?


I put the embedded path in a.out with:
cc -g -O0 -DUSE_DLOPEN -Wl,+b,\$ORIGIN:\$ORIGIN/../mydirB testlibfoo3.c

1) a.out did dlopen for libfoo.sl
2) dlsym to get symbol for routine foo
3) called foo in libfoo.sl

This is the code segment in libfoo.sl:

#ifdef USE_DLOPEN
printf("-- libfoo: do dlopen for libfoo2.sl \n" );
void *han = dlopen("libfoo2.sl", RTLD_NOW);
if ( !han ) {
printf("-- libfoo: dlopen of libfoo2.sl failed - %s \n", dlerror() );
assert(han != NULL);
}
else
printf("-- libfoo: dlopen of libfoo2.sl succeeded \n" );

was compiled, linked with,

cc -DUSE_DLOPEN -g -c +z libfoo.c
ld -b -o libfoo.sl +I init_foo libfoo.o



and output from a.out is:


- testlibfoo3: Version 0.14
- testlibfoo3: do dlopen for shared library libfoo.sl
-- libfoo: libfoo loaded
- testlibfoo3: using dlsym to get handle for libfoo
- testlibfoo3: call routine libfoo in libfoo.sl
-- libfoo: entered
-- libfoo: do dlopen for libfoo2.sl
-- libfoo: dlopen of libfoo2.sl failed - Can't open shared library: libfoo2.sl
Assertion failed: han != NULL, file libfoo.c, line 95
Abort (core dumped)

[howard rutiezer_1]

Ranganth,

I did man dlopen and man dld_getenv.
I can see dlopen but not dld_getenv.
Where is documentation for "dld_genenv"?
I don't see a man entry for "dld_getenv".
I see an example of the use of dld_getenv
in dlopen(3C) and no specification. I see
information on it in shl_load. Someone here
claims they tried, unsuccessfully, to use
dld_getenv, but I don't have a reproducer
for it.




Do I need MANPATH set or man pages installed?



Did you find any documentation on use
of shared libraries and $ORIGIN?

Does $ORIGIN work only in conjunction with
-l or -l: ?

Does libfoo.sl need an embedded pathname with $ORIGIN in it like a.out which chatr says looks like this:

shared executable
shared library dynamic path search:
SHLIB_PATH enabled first
embedded path enabled second $ORIGIN:$ORIGIN/../mydirB


If so, how do I get put it there?
I do this now for libfoo.c:



More information:


The last time I tried to draw a diagaram of
the directory structure, the spaces got
eliminated so it looks like the post on
5-Jan. Another attempt.

............. dlopen
............... |
...............ex12
................|
................/\
.............../..\
..........mydirA..mydirB
.............|.....|
............a.out libfoo.sl
..................libfoo2.sl




I ran a.out from various directories and
attempt to summarize the results I saw below. I get different results depending on the current working directory from which I run the a.out file.


Summary of current working directory,
command used, and result.


current
working
directory command result
--------- -------------------- ---------------
./dlopen ./ex12/mydirA/a.out crash as above
./ex12 ./mydirA/a.out runs OK
./mydirA ./a.out crash as above
./mydirB ../mydirA/a.out runs OK


You said in a prior post:

"At load-time (run-time), the loader will substitute
$ORIGIN" with the full path of the directory where the
current load module (executable or shared library) was
loaded from."


The a.out is in a fixed place, and I don't understand
why I see diffent results depending on the current working
directory when I run the program.


Thanks,
Howard

[ranganath ramachandra]

howard,

apparently something is wrong in the implementation of $ORIGIN on PA32. your program should have given you the 'OK' results in each of the 4 cases. i also noticed irregular behaviour with a similar program (where the dlopen is being done from with a shared library).

the documentation currently available for $ORIGIN is sparse. i am in the linker/loader team, i will file a defect on the project to make sure the behaviour and documentation are fixed (you can also do this yourself through the response center).

in the meanwhile: you can find documentation for dld_getenv in shl_load(3x) manpage.

you can use any of the following workarounds:

1. explicitly load shared libraries using dlopen as "$ORIGIN/../lib/libfoo.sl" and "$ORIGIN/libfoo2.sl" etc.

2. build your application with +s linker option (passed through compiler as -Wl,+s) and set SHLIB_PATH=\$ORIGIN/../lib in a startup script.

3. build the application with +s, and set SHLIB_PATH in the program itself :
putenv ("SHLIB_PATH=$ORIGIN/../lib");
and make the dld read it using
dld_getenv();

[howard rutiezer_1]

>in the meanwhile: you can find documentation for dld_getenv in shl_load(3x) manpage.

>you can use any of the following workarounds:

>1. explicitly load shared libraries using dlopen as "$ORIGIN/../lib/libfoo.sl" and "$ORIGIN/libfoo2.sl" etc.

>2. build your application with +s linker option (passed through compiler as -Wl,+s) and set SHLIB_PATH=\$ORIGIN/../lib in a startup script.

>3. build the application with +s, and set SHLIB_PATH in the program itself :
putenv ("SHLIB_PATH=$ORIGIN/../lib");
and make the dld read it using
dld_getenv();

Ranganath,

Thank you for your repsonse.

1) RE: dlopen and $ORIGIN error -- part of program seems "as if" the loader is using the directory from which the program is run rather than the directory in which the program being run resides.

2) Although I do not have a reproducer, a person here claims "dld_getenv" does not work correctly with "shl_load".

3) We will look at your other suggestions.

Thanks,
Howard