11.11 run level kill scripts not executed on shutdown

[David Mundfrom]

I managed to get the script provided from Oracle to work which restarts the databases and listener when our test server reboots. However I can't get the same script to execute on shutdown.

After much testing I am reasonably certain NONE of the scripts in /sbin/rc.d directories are called during shutdown.

I have tried different numbers (K100test, K800test, K050test, etc.), also each runlevel directory, nothing gets executed.

File /sbin/rc3.d/S990oracle links to /sbin/init.d/oracle and works perfectly on startup. However /sbin/rc2.d/K100oracle also links to the same script and does not execute on shutdown. I included "echo" commands which write to a logfile, during startup everything gets logged and during shutdown nothing.

I checked the syslog and /etc/rc.log, found no entries for shutdown.

Is there some other place the server logs shutdown?

Is there a kernel parameter which must be set for the server to go through the run-levels during shutdown?

I'm pretty happy to get the oracle stuff to restart automatically, and if I can get it to do an orderly shutdown I'll be absolutely ecstatic. It is a test server, so we are constantly tweaking it.

Any help would be greatly appreciated.



Here are my test links in the runlevel directories:

(permissions on the /tmp/test1.log is 777)

lrwxr-xr-x root sys rc0.d/K150test0 -> /sbin/init.d/test0
lrwxr-xr-x root sys rc1.d/K150test1 -> /sbin/init.d/test1
lrwxr-xr-x root sys rc2.d/K150test2 -> /sbin/init.d/test2
lrwxr-xr-x root sys rc3.d/K150test3 -> /sbin/init.d/test3
lrwxr-xr-x root sys rc4.d/K150test4 -> /sbin/init.d/test4

actual scripts:
-rwxr-xr-x root sys test0
-rwxr-xr-x root sys test1
-rwxr-xr-x root sys test2
-rwxr-xr-x root sys test3
-rwxr-xr-x root sys test4

contents of one of the scripts:
"test1" [Read only] 46 lines, 1044 characters
#!/bin/sh

PATH=$PATH:/bin:/usr/bin # FOR su

TESTLOG=/tmp/test1.log
echo " " >> $TESTLOG

case "$1" in
start)
echo "start $0 $1" >> $TESTLOG
date >> $TESTLOG
exit 0
;;
restart)
echo "restart $0 $1" >> $TESTLOG
date >> $TESTLOG
exit 0
;;
stop)
echo "stop $0 $1" >> $TESTLOG
date >> $TESTLOG
exit 0
;;
*)
echo "usage $0 $1" >> $TESTLOG
date >> $TESTLOG
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac
exit 0

[Patrick Wallek · Accepted solution]

Check your /etc/rc.log.old. That is the file that will contain information about shutdown.

How are you rebooting the server? Are you doing a 'shutdown -r' or are you doing a 'reboot'. The preferred method is via 'shutdown -r'.

If you are using 'reboot' then the K* scripts will NOT get executed. See the WARNING section of 'man 1m reboot' for details.

Have you tested the script by running it manually? Try running:

# /sbin/init.d/oracle stop

and see if it works.

[Steven E. Protter]

You have a script in /sbin/init.d/

You have links in /sbin/rc3.d for Start and /sbin/rc2.d for Kill.

Check rc.log to see if there is an error.

but a set -x in the /sbin/init.d script to get full diagnostics on the script as it runs.

My guess at this point is that the oracle environment is not loaded properly in your scripts and the scripts are executing but failing to do anything useful.

/usr/bin/su - oracle -c "command"

Should be used to switch to the oracle user for start/stop operations executed by root init process.

SEP

[Simon Hargrave]

I notice you are writing your test output to /tmp/test1.log etc.

Is CLEAR_TMP set to 1 in /etc/rc.config.d/clean_tmps? If so then your files will likely be written to, then deleted on startup.

Consider writing your logs to somewhere they won't get deleted, /var/adm for instance.

[Denver Osborn]

Was there anything on the console when the shutdown was run? If it wasn't shutdown from the console, does the console log have any info re: the shutdown? (login to GSP, CL to view console log)

I'd also look at the /etc/rc.log.old for anything that might help or show the last shutdown.

We are talking about a shutdown and not a reboot, right? :)

-denver

[renarios]

Hi David,

I see in your comment File /sbin/rc3.d/S990oracle links to /sbin/init.d/oracle and works perfectly on startup. However /sbin/rc2.d/K100oracle...

The kill script has to be in the same runlevel directory as the start script. i.e.
/sbin/rc3.d/S990oracle
/sbin/rc3.d/K110oracle

I hope that helps,

Renarios

[David Mundfrom]

You are correct, I was using the reboot command. I had also checked the /etc/rc.log.old file and found only shutdown messages.

I'll test again this weekend with the "shutdown -r" command.

I think that's probably where the problem lies.

Thanks also for the set -x tip, I'll use that, too.

Also, the CLEAR_TMP is not being executed, the same ancient files are still in our /tmp directory.

I'm pretty sure the oracle environment is fine, it works on startup.

It also works when I execute as root interactively. I do this:

1. "su - root" to make sure I don't still have the oracle user environment variables

2. set two env variable, ORA_HOME and ORA_OWNER

3. execute the script:
su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut"

And it works.

Here's the whole script (including tons of diagnostic messages).

#!/usr/bin/sh
PATH=/usr/sbin:/usr/bin:/sbin
export PATH
set -x
rval=0

LOG2=/home/oracle/logfiles/dbstarttest.log
echo " " >> $LOG2
echo "called /sbin/init.d/oracle $1" >> $LOG2
date >> $LOG2

# set ORA_HOME to be equivalent to the $ORACLE_HOME
# from which you wish to execute dbstart and dbshut
#
# set ORA_OWNER to the user id of the owner of the
# Oracle database in ORA_HOME

#ORA_HOME=/ora/product/8.1.7
ORA_HOME=/ora/product/9.2.0
ORA_OWNER=oracle

if [ ! -f $ORA_HOME/bin/dbstart ]
then
echo "oracle startup: cannot start"
echo "oracle startup: cannot start" >> $LOG2
exit
fi

case "$1" in
'start')
echo "you said start"
echo "you said start" >> $LOG2
#start the oracle databases
#the following command assumes the oracle login
#will not prompt the user for any values
#su - ${ORA_OWNER} -c $ORA_HOME/bin/dbstart &
LOG=/home/oracle/logfiles/autodbstart.log
echo " " >> $LOG
echo " " >> $LOG
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG
date >> $LOG
echo "$0: starting up" >> $LOG
su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart" >> $LOG
#su $ORA_OWNER -c $ORA_HOME/bin/dbstart &

echo "starting listener 9" >> $LOG
#$ORACLE_HOME/bin/lsnrctl start listener9>> $LOG 2>&1 &
#su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start listener9">> $LOG 2>&1 &
#su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start listener9" >> $LOG
export ORACLE_HOME=/ora/product/9.2.0
export PATH=/usr/bin:/usr/contrib/bin:/usr/local/bin:/usr/sbin:/$ORACLE_HOME
/bin
#export ORATAB=/etc/oratab
export TNS_ADMIN=$ORACLE_HOME/network/admin
su $ORA_OWNER -c "$ORACLE_HOME/bin/lsnrctl start listener9" >> $LOG

echo "starting listener 8" >> $LOG
export ORACLE_HOME=/ora/product/8.1.7
export PATH=/usr/bin:/usr/contrib/bin:/usr/local/bin:/usr/sbin:/$ORACLE_HOME
/bin
export TNS_ADMIN=$ORACLE_HOME/network/admin
su $ORA_OWNER -c "$ORACLE_HOME/bin/lsnrctl start listener8" >> $LOG 2>&1

rval=0
;;
'stop_msg')
echo "stop_msg" >> $LOG2
LOG=/home/oracle/logfiles/autodbstop.log
echo "stopping oracle databases" >> $LOG
;;
'stop')
echo "you said stop"
echo "you said stop" >> $LOG2
# stop the oracle databases
#su - ${ORA_OWNER} -c ${ORA_HOME}/bin/dbshut &
LOG=/home/oracle/logfiles/autodbstop.log
echo " " >> $LOG
echo " " >> $LOG
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG
echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOG
date >> $LOG
echo "$0: shutting down" >> $LOG
su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut" & >> $LOG
#su $ORA_OWNER -c $ORA_HOME/bin/dbshut &
rval=0
;;
*)
echo "usage dbstart|dbstop"
echo "usage dbstart|dbstop" >> $LOG2
rval=1
;;
esac

echo "script finished" >> $LOG2
date >> $LOG2

echo "script is finished"

[Patrick Wallek]

Renarios,

Actually, the way David is doing it is the preferred method. If you start something at run level X, then you typically stop it at run-level X-1.

If your default run-level is 3, then any K scripts in rc3.d would NOT get executed when the machine is shut down. The scripts get executed when transitioning INTO a run-level.

So if you are in run-level 3, and do a 'shutdown -r', then you are transitioning into run-level 2, and all K* scripts in rc2.d will get executed.

The only way for any K* scripts in run-level 3 to get executed is if you shutdown from run-level 4.

On my systems I have NO K* scripts in rc3.d.

[David Mundfrom]

Problem solved, it was indeed the "reboot" command (doing exactly what it's designed for) which caused the problem.

I used "shutdown -r 0" and voila - all kill scipts for run levels 2, then 1, then 0 were executed.

Thanks to everyone for your great suggestions and I'll post feedback first thing in the morning.

[David Mundfrom]

use shutdown -r command