oracle database instances not starting

[Rick Garland]

Hi all:

Working on L2000 with HP-UX 11.00 and Oracle 8.1.6

Have setup the rc start/shut scripts to start/stop the oracle instances on boot/halt. All looks good except the SIDs are not starting. Looking in the rc.log says they are starting but when it is checked, the SIDs are not running. A sample from the rc.log:


Start Oracle Database(s)
Output from "/sbin/rc3.d/S750oracle start":
----------------------------

Database "mimstst" warm started.

Database "nsmt" warm started.

Database "nsmp" warm started.


There are no errors in the rc.log related to the oracle start. The listener is starting and doing the check shows it is started. Not sure what is going on because the $ORACLE_HOME/bin/dbstart and dbshut are being called and these are standard with the oracle install. Where to go from here?

Many thanks!

[John Poff]

Rick,

I'd start by trying to run your oracle start script (/sbin/init.d/oracle or whatever) from the command line as root to see if you pick up any complaints. If that doesn't help, you'll have to try putting some extra "echo" statments in the dbstart script to see what is happening.

My guess is that the ORACLE_HOME and the PATH aren't getting set correctly and that it can't find 'svrmgrl'. Just a SWAG.

Is this a new implementation or were these scripts working and then just quit?

JP

[Rick Garland]

A new implementation. Did the /sbin/init.d/oracle start and the response back I get is the same as the rc.log file, the SIDs are warm started. But check the processes and they are not there.

Will put the echos in the script and double check the PATH and all.

[Rick Garland]

Checked the PATH and added the echos. No problems. $ORACLE_HOME is correct and doing the whence svrmgrl and whence sqlplus shows the commands are found.

The echos are showing the correct output.

A stumper no doubt.

[Joseph C. Denman]

hmmm. Not sure.

What gets me is the warm started. Your rc.log should contain all of the startup info.

Are you doing a
su - oracle -c "lsnrctl start;dbstart"
in your rc?

You should see the su in you rc.log?

...jcd...

[John Poff]

Rick,

I'm looking at a copy of the dbstart script for 8.1.6 here. It checks for the return code ($?) from sqlplus for version 8.1.6 when it does the startup. You only get the "warm started" message if you get a return status of zero, so it must be happy with what happens in 'sqlplus'. Either something strange is going on with sqlplus, or maybe the database is dying right after it starts?

JP

[Rick Garland]

Yes, the "warm started" is strange since I should see that it has done the connect, database mounted, etc...

Do not see the "su" messages but I am inclined to believe that I wouldn't. I have added the set -x to the /sbin/init.d/oracle script as well and the output looks good.

Could it be the $ORACLE_HOME/bin/dbstart script by chance?

[Rick Garland]

Part of what I see in the dbstart script is it is choosing between sqlplus or svrmgrl, checking the version, and depending on the version it does the start. However, what I also see is that the dbstart:version check doesn't look right. Anyone else think so as well?

[Joseph C. Denman]

I do not think that is it. However, I do not have 8.1.6. I do have 8.1.5 and 8.1.7 databases, and I have never had any problems with dbshut or dbstart.

Have you tried starting all instances from svrmgrl?

Don't know....very strange.

...jcd...

[John Poff]

Rick,

You won't see the 'su' command in their dbstart script as they do everything as root. To test that you could export your ORACLE_SID, ORACLE_HOME, PATH, and LD_LIBRARY_PATH just like they do in the script, and then try to run the sqlplus commmands manually. That might give some interesting clues.

JP

[Rick Garland]

Hate to say it, but "been there, done that".

Have tested all variables using the echo command and the whence command. All reports as expected and all can be found with no problems or errors to report.

Have watched the processes as I run the oracle start and the SIDs do not start. Have checked for the return on the command and I am getting $?=0. so it thinks all is OK. I can start the SIDs manually and everything is OK using that method but I am not about to revert to manual intervention for these servers.

[John Poff]

Rick,

Can you find any information in any kind of Oracle logs for the startup? The 8.1.6 dbstart script I have shows 'sqlplus /nolog' for the command. Is their a command switch for sqlplus to make it verbose?

The DBAs here always roll their own start/stop scripts. Now I see why!

JP

[Rick Garland]

I am going to look at the nolog option. But something else I found. I also have a new install (same HW and OS) with oracle 8.1.5 - and guess what, it works!

On the 8.1.6, ran the $ORACLE_HOME/bin/dbstart as oracle and what I see is what has been reported so far.

On the 8.1.5 I run the dbstart (as oracle) and the SIDs start! There is output indicating it connected, database mounted, sizes,etc., and then the warm start message.

[John Poff]

That's weird! I have 8.1.5 on this box so I did a 'diff' between the 8.1.5 dbstart and the 8.1.6 dbstart. The only difference is the LD_LIBRARY_PATH line, which the comment says was added for a bug.

JP

[Jim Turner]

Rick,

Please excuse the stupid question: Do you have a valid /etc/oratab file with your desired instances set to "Y" so they will start when your rc script runs?

Just Easter Eggin' . . . . .

Cheers,
Jim

[Rick Garland]

oratab has the SIDs set to Y.

Goin home now. Thinkin on rollin my own as well (dbstart that is)

[Volker Borowski]

Rick,

I did not figure out, if you do "su" or not.

If not, does root belong to the os-group that was linked to oracle upon install. It should be dba and/or oper as default.

If root is no os-group dba, the connect internal will not succeed at all. See sample below. Now what strikes is that the RC is 0 !
This might be confusing, esp. if the error messages are suppressed for some reason.

Do not know if this helps.
Volker


# id
uid=0(root) gid=1(other)
# ORACLE_SID=WOD
# ORACLE_HOME=/oracle/WOD
# export ORACLE_SID ORACLE_HOME
# /oracle/WOD/bin/svrmgrl <> connect internal
> startup
> EOF

Oracle Server Manager Release 3.0.6.0.0 - Production

(c) Copyright 1999, Oracle Corporation. All Rights Reserved.

Oracle8 Enterprise Edition Release 8.0.6.3.0 - Production
PL/SQL Release 8.0.6.3.0 - Production

SVRMGR> Password:
Password:
ORA-01031: insufficient privileges
SVRMGR> ORA-01031: insufficient privileges
SVRMGR>
Server Manager complete.
# echo $?
0
#

[Joseph C. Denman]

Rick,

I know the dbstart say it is warm started, but have you check your alert logs to see what oracle says is happening. The alert log will be in the background_dump_dest directory. You can find this in your init file.

...jcd...

[John Poff]

Rick,

Just one more idea, try copying the 'dbstart' script from the 8.1.5 bin directory and running it to start the 8.1.6 instance. If that script fails also, I'd suspect something in the oracle setup - permissions, environment variables, etc. If that script works, it is a scripting problem with the 8.1.6 script.

At least that might eliminate one variable.

JP

[Pedro Sousa]

Hi Rick!
I hope this explains/solves your problem.
from: http://forums.itrc.hp.com/cm/QuestionAnswer/1,1150,0x0c0c6d96588ad4118fef0090279cd0f9,00.html

The following is from MetaLink:

Problem Description
===================

"dbstart" is a shellscript used on Unix platforms to start databases
listed in the "oratab" file. It is often called from system startup
scripts.

The "dbstart" script shipped with all versions of Oracle up to and
including 8.1.6 does not start any 8.1.6 instances if they are down.
"dbstart" will report "Database warm started" rather than starting
the instance. This problem only affects 8.1.6 (or later) instances.

The change in the banner may affect users custom startup / shutdown
scripts.

"dbshut" has no problem.

Please note there are 2 related issues which should be born in mind
when changing any startup / shutdown scripts:

Advance notification of Server Manager desupport [NOTE:74943.1]
Advance notification of CONNECT INTERNAL desupport [NOTE:50508.1]


Solution Description
====================

Enterprise Edition Workaround
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The workaround is to change the dbstart script to look for the string
"Oracle8i Enterprise Edition" instead of "PL/SQL" by making the following
changes:

1. Save a copy of the 8.1.6 $ORACLE_HOME/bin/dbstart script

2. Edit the script and change the 'awk' lines
from:
VERSION=`$ORACLE_HOME/bin/svrmgrl command=exit | awk '
/PL/SQL (Release|Version)/ {substr($3,1,3) ;
print substr($3,1,3)}'`
to:
VERSION=`$ORACLE_HOME/bin/svrmgrl command=exit | awk '
/PL/SQL (Release|Version)/ { print substr($3,1,3); exit }
/Oracle7 Server (Release|Version)/ { print substr($4,1,3); exit }
/Oracle8i Enterprise Edition (Release|Version)/ { print substr($5,1,3); exit }
/Oracle8i (Release|Version)/ { print substr($3,1,3); exit }'`

Additionally for platforms which set LD_LIBRARY_PATH in the dbstart
script you should change the following:
From:
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${ORACLE_HOME}/lib ;
export LD_LIBRARY_PATH
To:
LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH} ;
export LD_LIBRARY_PATH


3. Copy the updated "dbstart" to any location it is expected to reside.
Eg: system startup scripts often run /usr/local/bin/dbstart.

IMPORTANT: Do not overwrite existing scripts without taking a backup
copy first as the script you are overwriting may have
been modified.

NOTE: This workaround will only work for Oracle8 Enterprise Editions
Similar workarounds may be used for other editions.


Explanation
-----------

The problem is caused by a change in the Oracle banner which is output
by Oracle version 8.1.6. The banner no longer includes a PL/SQL line
and so the test in "dbstart" for a PL/SQL version returns a blank string
which in turn causes the step to startup the instance to be skipped.


References
----------

The problem is reported in [BUG:1154931]

The "oratab" file typically exists in either /etc/oratab or
/var/opt/oracle/oratab

[NOTE:50508.1] - Advance notification of CONNECT INTERNAL desupport
[NOTE:74943.1] - Advance notification of Server Manager desupport

hope this helps.
good luck.

[Hamdy Al-Sebaey]

Hi Rick,
To start the listener,try this;
su - oracle,
#home/oracle>lsnrctl start.

Also U can stop &start your DB in archive mode;
#su - oracle
#ome/oracle>svrmgrl
#svrmgr>connect internal
#svrmgr>alter system archive log stop
#svrmgr>alter system archive log start
#svrmgr> exit
#home/oracle>sqlplus system/11std653@(the name of your DB)
#sql>exit
#home/oracle>exit
I hope it'll help;
Good luck
Hamdy