Start/stop of Oracle Database not wokring from <sid>adm (SAP User)

[Sanjay Kumar Suri]

We have done R3copy from a live production system to a test system with a new sid and hostname.

Both the system have the same version of Oracle(9.2.0.5) and HP-UX (11.11).

Now are able to start Oracle from ora. However it is not starting using adm and give the following errror in startdb.log:

/usr/lib/pa20_64/dld.sl: Unable to find library 'libclntsh.sl.9.0'.
/usr/sap/QEW/SYS/exe/run/startdb[228]: 12835 Killed

Similarly while trying to stop SAP/Oracle from adm we are getting the following error in stopdb.log:

/usr/lib/pa20_64/dld.sl: Unable to find library 'libwtc9.sl'.
/usr/sap/QEW/SYS/exe/run/stopdb[2]: 12306 Killed

Urgent help will be highly appreciated.

sks

[Indira Aramandla]

Hi SKS,

This could be because you do not have any reference to $ORACLE_HOME/lib in your LD_LIBRARY_PATH or any links to $ORACLE_HOME/lib, so the command cannot find libclntsh.sl.9.0.

You can either set your LD_LIBRARY_PATH or link to your $ORACLE_HOME/lib from
/usr/lib.

Then create a link from the libclntsh alias under /usr/lib to the libclntsh file in the $ORACLE_HOME/lib directory.

ln -s $ORACLE_HOME/lib/libclntsh.so.1.0 /usr/lib/libclntsh.so.1.0

check the ldd command

I hope this helps

Indira A

[Eric Antunes]

Hi Sanjay,

See this thread on Metalink:

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=FOR&p_id=504521.994

Best Regards,

Eric Antunes

[Yogeeraj_1]

hi,

below a quote from man ld:


Some restrictions exist for 64-bit mode with executables built with shared libraries and with S-bit set (through +s enabled). The behavior of 64-bit executables does not match the behavior of equivalent 32-bit executables or executables built with +compat. For any setuid or setgid programs, dld disables any dynamic library searching through environment variables, SHLIB_PATH for 32-bit and 64-bit mode, and LD_LIBRARY_PATH for 64-bit mode. If a library only exists in the directory specified in SHLIB_PATH (or LD_LIBRARY_PATH),
you get the runtime error "library not found" if the program is a setuid program (that is, uid not equal to euid or gid not equal to
eguid) and it depends on that library. dld uses the dynamic path lookup (with SHLIB_PATH) only if the following conditions are satisfied: getuid () == geteuid () && getgid () == getegid (). That is, if the uid or gid does not match its effective counterpart, dld searches only the recorded library path. As a result, dld does not check the SHLIB_PATH which causes the runtime error "library not found".



this can be related...

hope this helps too!

regards
Yogeeraj

[Fred Ruffet]

Hi,

Your adm user must have $LD_LIBRARY_PATH with $ORACLE_HOME/lib included in his env.
A line like this in his profile sould be ok :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

Regards,

Fred

[Sanjay Kumar Suri]

Hello Indira

As suggested we did the following:

1. Defined LD_LIBRARY_PATH in .dbenv_.csh
setenv LD_LIBRARY_PATH $ORACLE_HOME/lib

2. cd /usr/lib
ln -s $ORACLE_HOME/lib/libclntsh.so.1.0 /usr/lib/libclntsh.so.1.0

Now when starting Oracle database the message is:

/usr/lib/pa20_64/dld.sl: Unable to find library 'libodm9.sl'.

We again did the ln -s for 'libodm9.sl' but same error. Any more clues?

We are not sure on ldd command.

Dear Eric: We don't have Oracle support. Can you download and send the page as an attachment.

Dear Yogi: Not able to process your reply. Need more inputs.

Dear Fred: Thanks.


sks

[Fred Ruffet]

Making links for all libs may not be a solution cause there are really a lot...

What can be done is to issue the command
ldd $ORACLE_HOME/bin/oracle
with oracle user and adm user.

First one will resolve all dependances, whereas second will show you missing dependances. Adding pathes to adm $PATH var will probably solve the problem.

Regards,

Fred

[Sanjay Kumar Suri]

Hello Fred

Using the adm user (ldd output is)
ldd oracle
Unable to find library 'libodm9.sl'.

What next?


sks

[Fred Ruffet]

You said you made a copy of Oracle from a server to another... On the same PATH ? Did you relink oracle kernel ?

if not, stop all running DBs, and issue "relink all" connected as oracle user.

Regards,

Fred

[Eric Antunes]

Hi Sanjay,

Here is the thread in attach.

I really hope this will help you but, reading it I didn't understood much...!

Eric

[Eric Antunes]

Sanjay,

You did a copy or did you cloned the database? I'm asking this because you can't copy 2 instance with differents SID, you must clone them.

Best Regards,

Eric Antunes

[Indira Aramandla]

Hi Sanjay,

Sorry for the delayed repy. Attached is a document which describes how HP-UX loads shared libraries at runtime.

This document discusses setting SHLIB_PATH and LD_LIBRARY_PATH on HP-UX. However, even with the correct settings of these environment variables, dynamic loading of shared libraries can still fail

Any other library linked with the executable could cause the error, such as:
libwtc8.sl
libobk.sl
libodm9.sl
libjox9.sl

I hope this will fix the issue.


Indira A

[Volker Borowski]

Hi sks,

was this QEW system ever up and running before and you simply exchanged the database, or did you set up an entirely new QA-box ???

If entire new, how did you set up the rest of the system ? I think of an environment mismatch for your sidadm.

Can you attach the output of
"env"
for both users pleas and tell us what SAP-version you are running.

Thnaks
Volker

[Sanjay Kumar Suri]

Dear all

Thanks for so many inputs and the problem is as it is. Let me narrate what we did:

Source System
---------------------
1. We took entire offline FS backup from a running SAP system (SID=ABC, hostname=host1).

Target System
--------------------

1. This backup was restored.

2. SAP was started as it is with SID=ABC and hostname=host1 using user=abcadm. And no issues were faced.

3. Using Oracle command:
create controlfile reuse set database XYZ

We changed the SID of target system to XYZ.

4. We also made sutiable changes in initXYZ.ora and other changes in files tnsnames.ora, listener.ora etc.

5. Now we changed the host name to host2.

6. We renamed profile for xyzadm and oraxyz users to .dbenv_host2.csh and .sapenv_host2.csh and made required changes.

7. Also recreated links in $ORACLE_HOME/lib, $ORACLE_HOME/lib32 and /usr/lib to reflect new host name and SID.

Now we are not able to start Oracle using xyzadm. But we can start the same with oraxyz.

There error faced is shown below:

host2:xyzadm 28> sqlplus /nolog
/usr/lib/pa20_64/dld.sl: Unable to find library 'libwtc9.sl'.
Killed

host2:xyzadm 29> tnsping
/usr/lib/pa20_64/dld.sl: Unable to find library 'libclntsh.sl.9.0'.
Killed

sks

[Eric Antunes]

Hi Sanjay,

"Now we are not able to start Oracle using xyzadm. But we can start the same with oraxyz."

Can you post here "ev|more" outputs for those 2 users?

Best Regards,

Eric Antunes

[Eric Antunes]

I mean env, not ev.

[Sanjay Kumar Suri]

The enviroment seems to be ok. Still:

host2:qewadm 22> env
HOME=/home/qewadm
PATH=/oracle/QEW/920_64/bin:.:/home/qewadm:/usr/sap/QEW/SYS/exe/run:/usr/bin:/usr/local/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/hparray/bin:/opt/nettladm/bin:/opt/upgrade/bin:/opt/fcms/bin:/opt/pd/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/mx/bin:/opt/resmon/bin:/opt/netscape:/opt/scr/bin://opt/perl/bin:/opt/graphics/common/bin:/usr/sbin/diag/contrib:/opt/ignite/bin:/opt/omni/bin
LOGNAME=qewadm
SHELL=/usr/bin/csh
TERM=dtterm
COLUMNS=134
LINES=49
MANPATH=/usr/share/man/%L:/usr/share/man:/usr/contrib/man/%L:/usr/contrib/man:/usr/local/man/%L:/usr/local/man:/opt/upgrade/share/man/%L:/opt/upgrade/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/ignite/share/man/%L:/opt/ignite/share/man:/opt/mx/share/man:/usr/dt/share/man:/opt/samba/man:/opt/resmon/share/man:/opt/scr/share/man://opt/perl/man:/opt/graphics/common/man:/opt/hparray/share/man/%L:/opt/hparray/share/man:/opt/omni/lib/man
TZ=IST-5:30
SAPSYSTEMNAME=QEW
DIR_LIBRARY=/usr/sap/QEW/SYS/exe/run
SHLIB_PATH=/usr/sap/QEW/SYS/exe/run
THREAD=NOPS
dbms_type=ORA
dbs_ora_tnsname=QEW
dbs_ora_schema=SAPR3
ORACLE_PSRV=QEW
ORACLE_SID=QEW
DB_SID=QEW
ORACLE_HOME=/oracle/QEW/920_64
ORACLE_BASE=/oracle
ORA_NLS33=/oracle/client/81x_64/ocommon/nls/admin/data
NLS_LANG=AMERICAN_AMERICA.WE8DEC
SAPDATA_HOME=/oracle/QEW

host2:oraqew 21> env
HOME=/oracle/QEW
PATH=/oracle/QEW/920_64/bin:/oracle/QEW/bin:/usr/sap/QEW/SYS/exe/run:/usr/bin:/usr/local/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/hparray/bin:/opt/nettladm/bin:/opt/upgrade/bin:/opt/fcms/bin:/opt/pd/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/mx/bin:/opt/resmon/bin:/opt/netscape:/opt/scr/bin://opt/perl/bin:/opt/graphics/common/bin:/usr/sbin/diag/contrib:/opt/ignite/bin:/opt/omni/bin
LOGNAME=oraqew
SHELL=/usr/bin/csh
TERM=vt100
COLUMNS=134
LINES=49
MANPATH=/usr/share/man/%L:/usr/share/man:/usr/contrib/man/%L:/usr/contrib/man:/usr/local/man/%L:/usr/local/man:/opt/upgrade/share/man/%L:/opt/upgrade/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/ignite/share/man/%L:/opt/ignite/share/man:/opt/mx/share/man:/usr/dt/share/man:/opt/samba/man:/opt/resmon/share/man:/opt/scr/share/man://opt/perl/man:/opt/graphics/common/man:/opt/hparray/share/man/%L:/opt/hparray/share/man:/opt/omni/lib/man
TZ=IST-5:30
SAPSYSTEMNAME=QEW
dbms_type=ORA
DIR_LIBRARY=/usr/sap/QEW/SYS/exe/run
SHLIB_PATH=/oracle/QEW/920_64/lib
DBENV_CSH=SET
NLS_LANG=AMERICAN_AMERICA.WE8DEC
THREAD=NOPS
ORACLE_PSRV=QEW
ORACLE_SID=QEW
ORACLE_HOME=/oracle/QEW/920_64
ORA_NLS33=/oracle/QEW/920_64/ocommon/nls/admin/data
SAPDATA_HOME=/oracle/QEW
ORACLE_BASE=/oracle

sks

[Yogeeraj_1]

hi,

The problem may be arising because the programs are setuid and setgid. This turns off library path search see man ld.

you should check your environment variables: SHLIB_PATH and LD_LIBRARY_PATH

regards
Yogeeraj

[Eric Antunes]

Hi Sanjay,

I suspect your variable ORA_NLS33 (/oracle/client/81x_64/ocommon/nls/admin/data) in not ok for qewadm user...

Since your are on 9i, I think (not sure because I'm still on 8i) you shouldn't use 8i nls.

Best Regards,

Eric Antunes

[Sanjay Kumar Suri]

Dear Friends

Thanks for all the posts.

We solve the problem as suggested by the inputs from Fred using ora.

$relink all

Fred - I need to give 10 point to you whenever you reply to this thread.

Indira - You attached document is really good. We will share the same with our team members.

sks

[Fred Ruffet · Accepted solution]

hmmm ? Talking to me ? ;-)

As I need to add value to the thread (not only looking for points, noooo !) :
For the next time you will do this kind of operations, have a look at Oracle utility called nid in $ORACLE_HOME/bin (dbnewid). It changes a DB Name or DB ID in a more easy and secure way than using create database order.

Regards,

Fred

[Volker Borowski]

Hi,

glad you fixed it !

Watch it, with this procedure it is NOT supported to apply any further bugfixes or patches to Oracle-SW, because the ORACLE_HOME is spread throughout the entire Oracle-Inventory in ascii and binary files.

There are OSS Notes decribing the problem.

If you have to patch after a ORACLE_HOME rename, you need to delete the entire Oracle SW AND the inventory and do a reinstall of the software do get a correct inventory!
This will takes only some 20-40 minutes.

If you stay on the SW without applying patches/bugfixes, no problem !
I'd recommend to insert a note to the login-messages for orasid, just in case you need to patch in a year or so. Otherwise you might pull out your hair for inventory-mismatch-problems.

Volker

[Lourdes Ortiz Sada]

Hi Sanjay

Im see that you have one error similar that I have in this moments...

Could you send me the document that Indira sent to you?

Best Regards =)

[Yogeeraj_1]

hi lourdes,

just click on the "attachments" icon found to the left of the post (indira's) and download it from here itself.

regards
yogeeraj

[Sanjay Kumar Suri]

Hello Sada

Please find attached is another relevant SAP Note.

sks