Can't open shared library

[bivian]

If the loader gives following error
"Can't open shared library: ..."
what does it indicate?

The shared library has been linked with +s and +b options. The SHLIB_PATH has been set properly.
Is there anything else that is missing?

[Dennis Handly · Accepted solution]

>"Can't open shared library:" what does it indicate?

Does it give a reason?

>Is there anything else that is missing?

Have you done the same to the executable?
What HP-UX version and architecture are you using?

[bivian]

No i don't see any reason.

I have done the same thing for executable also.

I am HPUX PARISC64. Following are details from uname command:
"HP-UX B.11.23 9000/800"

[bivian]

Just to add the executable and the shared library are both linked on one machine and are being used in another machine.

[Dennis Handly]

>No I don't see any reason.

No errno text in the message? What was the complete message?

What does "ldd executable" show?
You may have to use "tusc -fp -ea -o tusc.out executable" to trace what's happening.

[bivian]

I am not sure which executable is trying to load the shared library. The error message "Can't open shared library libclntsh.so" is logged into the log files.

tusc command shows some useful information.

From tusc.out i see that it finds the library in one location and open call succeeds. But later the open on the same library fails. tusc.out shows something like this:

.....
.....
[4268] open("/home/prash/lib32/libclntsh.sl", O_RDONLY, 0200) =
[4268] fstat(3, 0x77ff4748) .............................. = 0
[4268] read(3, "0210010e0512@ \0K i 5 s \0\0\0\0".., 128) = 128
[4268] lseek(3, 128, SEEK_SET) ........................... = 128
[4268] read(3, "10\0\004\0\0\0( \0c + d4\0\010\0".., 48) . = 48
[4268] read(3, "80\0\0\v\0\0\004\0\0\0\0", 12) ........... = 12
[4268] mmap(NULL, 6500352, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 1757184
) = 0xcd400000
[4268] mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOU
S|MAP_SHLIB, -1, 0) = 0x77ddd000
.....
....

[4336] open("/home/prash/lib32/libnmemso.sl", O_RDONLY, 0200) =
[4334] close(3638) ....................................... ERR#9 EBADF
[4336] fstat(3, 0x77ff49c8) .............................. = 0
[4334] close(3639) ....................................... ERR#9 EBADF
[4336] read(3, "0210010e0512@ \0K i 5 s \0\0\0\0".., 128) = 128

[bivian]

The second entry in tusc.out is also for libclntsh.so...

[ranganath ramachandra]

Please attach the complete tusc trace that you get from
tusc -s open,read,close,mmap,munmap,stat,fstat -fp -ea -o tusc.out executable

[Dennis Handly]

>I am not sure which executable is trying to load the shared library.

It should be obvious. You should know your application. And tusc will tell you:
tusc -fp -ea -o tusc.out application ...
(Are you using threads?)

>But later the open on the same library fails.

This isn't really later on, this is a separate process. And the shlib is different.

>[4268] open("/home/prash/lib32/libclntsh.sl", O_RDONLY, 0200) =

(Please make sure you provide the return values.)

[4336] open("/home/prash/lib32/libnmemso.sl", O_RDONLY, 0200) =
[4334] close(3638) . ERR#9 EBADF
[4336] fstat(3, 0x77ff49c8) . = 0
[4334] close(3639) . ERR#9 EBADF
[4336] read(3, "0210010e0512@
> The second entry in tusc.out is also for libclntsh.so.

Huh? It says libnmemso.sl.

Are these PIDs or thread IDs?

[bivian]

Sorry for the delayed reply..

I have run the command "tusc -s open,read,close,mmap,munmap,stat,fstat -fp -ea -o tusc.out" on the process id.
It gives out a huge file which i cannot paste here.
Can you please tell me what parts of the file i need to paste.

[Dennis Handly]

>It gives out a huge file which i cannot paste here.

Have you tried gzip on it?

>Can you please tell me what parts of the file I need to paste.

Basically only the process with the error. And only the parts where the error occurs and enough history to see what's going on.

[bivian]

Thanks.
I did not realize that we can attach files. I have zipped and attached the output file.

Btw, please ignore the error messages that i mentioned above...The following is the error logged in error logs of application:

Could not load library '/home/prash/newdb/lib32/libnmcfsga.sl' for reason 'Can't open shared library: libnmemso.sl'

[Dennis Handly]

>The following is the error logged in error logs of application:
>Could not load library '/home/prash/newdb/lib32/libnmcfsga.sl' for reason 'Can't open shared library: libnmemso.sl'

I see that several times. libnmemso.sl must be a dependent shlib with no particular path??

Please provide the output of "chatr /home/prash/newdb/lib32/libnmcfsga.sl".

The processes 28914, 28929, 28954 & 28958 are using libjvm.sl. Which dlopens(?) /home/prash/newdb/lib32//libnmemso.sl.

Is this the one you want?
Also, it would be helpful if you also included the exec system call:
-s open,read,close,mmap,munmap,stat,fstat,exec

[bivian]

Chatr output of libnmcfsga.sl:
libnmcfsga.sl:
shared library
shared library dynamic path search:
SHLIB_PATH enabled first
embedded path disabled second Not Defined
shared library list:
dynamic libnmemso.sl
dynamic libjava.sl
dynamic libjvm.sl
dynamic /usr/lib/libcl.2
dynamic /usr/lib/librt.2
dynamic /usr/lib/libpthread.1
dynamic /usr/lib/libnss_dns.1
dynamic /usr/lib/libdld.2
dynamic /usr/lib/libnsl.1
dynamic /usr/lib/libm.2
shared vtable support disabled
explicit unloading enabled
runtime checks disabled
static branch prediction disabled
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation disabled
text segment locking disabled
data segment locking disabled
third quadrant private data space disabled
fourth quadrant private data space disabled
third quadrant global data space disabled
data page size: D (default)
instruction page size: D (default)
nulptr references disabled
shared library private mapping disabled


Yes, libnmemso.so belongs to the directory /home/prash/newdb/lib32/. This is the one that is not loading.

Please see the attached tusc1.out which contains exec system call also.

[Dennis Handly]

>SHLIB_PATH enabled first
>shared library list: dynamic libnmemso.sl

This says that if SHLIB_PATH contains m/home/prash/newdb/lib32, it should work.

What is the tusc command you are using? Why do we see other processes besides 25326, emagent? Can we start tusc on emagent and not attach after it starts?
Is emagent a SETUID process?
Why does 25326 keep going after the error? (We only need to see the error once.)
Is SHLIB_PATH before emagent starts?

>tusc1.out which contains exec system call also.

Please also include fork & exit.

[bivian]

I was using the command - tusc -s open,read,close,mmap,munmap,stat,fstat,exec -ea -o tusc.out" on the process id.

No, it is not SETUID process.

I have use the following command now:
tusc -fp -ea -o tusc.out
Please see the uploaded output in tusc2.out.

[Horia Chirculescu]

Hello,

stat("/usr/lib/libnmemso.sl", 0x778410d0) ........ ERR#2 ENOENT


You could make a soft link of the library in the /usr/lib directory

ln -sf /home/prash/newdb/lib32/libnmemso.sl /usr/lib/libnmemso.sl


Or... maybe you have some problem with SHLIB_PATHS ? (the path to the libnmemso.sl library is listed three times, should be only one).

SHLIB_PATH=/home/prash/newdb/jdk/jre/lib/PA_RISC2.0/hotspot:/home/prash/newdb/jdk/jre/lib/PA_RISC2.0:/home/prash/newdb/lib32:/home/prash/newdb/network/lib32:/home/prash/newdb/perl/lib:/home/prash/newdb/lib32:/home/prash/newdb/oui/lib/HP-UX:/home/prash/newdb/lib32

Horia.

[Dennis Handly]

>Please see the uploaded output in tusc2.out.

Unless you can get your /home/prash/newdb/bin/emagent process to print out its SHLIB_PATH, we can only assume it doesn't have /home/prash/newdb/lib32?

Note: SHLIB_PATH must be set before emagent starts, not just before the dlopen/shl_load.

What linker/dld patches do you have?

[bivian]

I will investigate whether the SHLIB_PATH is being set or not.

The linker patch i am using is phss_39821.

[Dennis Handly]

>The linker patch I am using is PHSS_39821.

That is the recommended patch, there is a later one: PHSS_40537

[bivian]

I am setting SHLIB_PATH properly. It does not work even if i create the soft links to libraries under "/usr/lib".

Is there a known issue with PHSS_39821?

I will try to install PHSS_40537.

[bivian]

Installed patch PHSS_40537. But still there is a problem.

[Dennis Handly]

>It does not work even if I create the soft links to libraries under "/usr/lib".

Doh, that will catch the culprit red handed and removes SHLIB_PATH settings from any problem.
Now please provide a tusc output to see why the open fails.
Can you do:
file /home/prash/lib32/libnmemso.sl
file /home/prash/newdb/bin/emagent

Hmm, tusc says the latter is 32 bit.
And the former has /usr/lib/libc.2.

>Is there a known issue with PHSS_39821?

Not that I know of.

[bivian]

It seems that i missed out few libraries while adding soft links to "/usr/lib". After adding all of them it works fine. But, this is just a work around. I am curious to know why SHLIB_PATH is not working.

> file libnmemso.sl
libnmemso.sl: PA-RISC1.1 shared library -not
> file emagent
PA-RISC1.1 shared executable dynamically linked -not stripped dynamically linked

> file /usr/lib/libc.2
PA-RISC2.0 shared library -not stripped

[Dennis Handly]

>I am curious to know why SHLIB_PATH is not working.

I don't think we'll ever know until you get out your debugger. And be able to follow the process from the start.
The only trouble areas I know are:
1) SHLIB_PATH not set correctly before process start.
2) SETUID processes.