Oracle startup/shutdown on HP-UX 11.31

[Gary Lanzi]

Server: HP-UX 11.31 on rx3600(IA)
Oracle 11

On past server installs using earlier versions of HP-UX we were able to place a start link at the end of /sbin/rc3.d and a stop link at the beginning of /sbin/rc2.d, with the links pointing to a script in /sbin/init.d, to gracefully shut down and startup Oracle on a server re-boot.

This is our first use of 11.31 and the process does not work for startup of Oracle. In looking at /etc/rc.log it looks like the link at the end of run level 3 calls the script, but the OS seems to finish booting and cuts off the script from starting the databases. The oracle shutdown script does work fine when called as the server processes run level 2 scripts on server shutdown. Normal run level for these servers is run level 3. Our Oracle DBA can successfully run these scripts from command line while logged in. We have new IA server installs and a few existing PA-RISC servers being upgraded to 11.31 that all have this problem.

I'd be grateful to anyone who can share any knowledge on this subject.

Thanks
Gary Lanzi

[Bill Hassell]

> but the OS seems to finish booting and cuts off the script from starting the databases.

Appearances can be deceiving...

Start with your start/stop script: are there any background processes (ie, & at the end of selected processes)? To see what the script is doing, turn on script tracing by adding set -x at line 2. The results will be logged in rc.log. Something probably went wrong in the script but no logs or error messages are present in the script.

[Dennis Handly]

>start link at the end of /sbin/rc3.d and a stop link at the beginning of /sbin/rc2.d, with the links pointing to a script in /sbin/init.d,

What are your exact names? Are they in the right order for /etc/rc.log?

>cuts off the script from starting the databases.

How do you know that? How much logging do you do?

[TwoProc]

Gary, just make sure that your script runs last in the /sbin/rc3.d.

I usually create an S999rc.local script that starts up whatever I want after everything else is up. If it's a timing issue, put a sleep command in there for a minute or two, so that everything is settled down.

I know that this is no long the "approved" method, but its what I still use for my big "runlast" command when a system comes up.

If you don't want to use the the whole rc.local idea - the just put your startup script at /sbin/rc3.d/S999Oracle so it's last to come up. Once again, use a sleep command if necessary.

Also, I disagree with your technique of putting the stop script in rc2.d, while the start is in rc3.d. I would make sure that the whatever I started up last, should go down first. That is, if you put a startup command in /sbin/rc3.d/S999Oracle you should have a matching /sbin/rc3.d/K999Oracle to go with it. Reason: Do you really want your Oracle DB to remain up while other system functions are going down (everything that starts with a K in level 3)? No, too risky, bring down the Oracle DB first, that way anything that should be up while the database is up would be available for the database the whole time.

Most times, I've seen sysadmins do this to so that an application depending on Oracle comes down first, before the database. If this is your case, handle this issue with the numeric ordering in the level 3, not with two different levels. That is, IMHO, Oracle DB is definitely a level 3 start/stop, not a level 2, so that level 3 has your application that uses Oracle DB as a dependency. If this is your case, then an example would be Oracle having start and stop of 998 and your application at 999. Of course use high numbers that you want, but you get the idea here.


Lastly, to make this posting really long - your problem may not be related to run levels. I've seen before, between Oracle versions, for whatever reason, suddenly the Oracle database needs/wants attached I/O different than the last version (or different version) you were using.

To test the above out, have your DBA try the commands from an "at" command and see if they still work. If they don't - then this is your problem, not run your init start and stop scripts. The difference being: Can your DBA start/stop Oracle when he/she is "not logged in", independent of whether or not your system is coming up or down.





That way,

[James R. Ferguson]

Hi gary:

With regard to the order and run-levels at which start and kill scripts should be specified, remember that kill scripts for a start script in directory '/sbin/rc(n)' will reside in '/sbin/rc(n-1)'.

The value of 'n' should be a three-digit number preceded by 'S' for startup ones and 'K' for kill scripts. The "rule-of-1000" should be followed and if applied, enables scripts with dependencies to be started and stopped while meeting their dependency relationships. That is, if a startup script is lined into '/sbin/rc3.d/S900mything' then the corresponding kill script would be '/sbin/rc2.d/K100mything'.

The manpages for 'rc(1M)' document this paradigm as well as the classic whitepaper, "HP-UX 10.x Startup and Configuration" available in the HP document archive:

http://docs.hp.com/en/archive.html

Regards!

...JRF...

[nightwich]

Hi Jonh

I perfectly agree with Jonh.

All good practices says tha the rule is rc(n-1).

Although you can have S999mything in Sbin/rc3.d, and K999mythink in the /sbin/rc2.d.

And the /sbin/rc2.d/K999mythink be the first shutdown script to run. Because usually the shutdown scripts start's at level rc2.d.

In the level 3 /sbin/rc3.d usually you should only have startup scripts and at level 0 /sbin/rc0.d you sould have only shutdown scripts.


Best regards.

[Gary Lanzi]

I'm still working this issue but want to let the forum know I'm keeping up with the posts and am grateful to everyone taking the time to respond.

John, Your answer on run levels seems to be off. See the the response from James, which is the way I've always known the sequencer to work.

James, Thanks for the info. I'm in the process of reading the rc manpage and have saved a pdf of the white paper from the link you attached for further reading.

For some of the questions I've been asked...
On exisitng servers prior to our 11.31 upgrade project, we always put in an /sbin/rc.2/K100oracle and an /sbin/rc3.d/S999oracle link and those have worked great for us. On a new IA server with a bare metal install of 11.31, with the same links, the K100 link seems to run fine, while the S999 link does not start the db's, which is evident as soon as I log into the server and check for Oracle processes. While I can find the most recent sequence of startup events in /etc/rc.log, to find the most recent sequence of shutdown events I normally look in /etc/rc.log.old. From reading other posts, I also am sure to check that the /etc/rc.config.d/oralce file
is there and has the proper syntax.

Here is an exerpt from a server rc.log file that shows no indication of startup errors. I am not familiar with the set -x command described by Bill, but will try adding that if it will show a more detailed output than what I pasted here. I'd also like to try adding a delay, but not sure where to do that.

Start LVM daemon
Output from "/sbin/rc3.d/S900lvm start":
----------------------------
Starting the amgrd subsystem
Output from "/sbin/rc3.d/S998amgrd start":
----------------------------
"/sbin/rc3.d/S998amgrd start" SKIPPED
usage: /sbin/rc3.d/S999oracle {start|stop}
Output from "/sbin/rc3.d/S999oracle start":
----------------------------
+ ORACLE_HOME=/u/app/oracle/local
+ ORACLE=oracle
+ PATH=/sbin:/u/app/oracle/local/bin
+ HOST=server03
+ + uname
PLATFORM=HP-UX
+ export ORACLE_HOME PATH
+ [ ! = ORA_DB ]
+ [ HP-UX = HP-UX ]
+ /usr/bin/su oracle -c /sbin/rc3.d/S999oracle start ORA_DB
+ ORACLE_HOME=/u/app/oracle/local
+ ORACLE=oracle
+ PATH=/sbin:/u/app/oracle/local/bin:/u/app/oracle/local/bin
+ HOST=server03
+ + uname
PLATFORM=HP-UX
+ export ORACLE_HOME PATH
+ [ ! ORA_DB = ORA_DB ]
+ /u/app/oracle/local/bin/oracle.startup /u/app/oracle/local
+ exit
+ exit

Starting PRNGD (Pseudo Random Number Generator Daemon)
Output from "/sbin/rc3.d/S99prngd.rc start":
----------------------------
"/sbin/rc3.d/S99prngd.rc start" SKIPPED

Starting Control Daemon
Output from "/sbin/rc3.d/S99z0OVCtrl start":
----------------------------
"/sbin/rc3.d/S99z0OVCtrl start" SKIPPED

**************************************************
HP-UX run-level transition completed
Fri Jul 9 16:50:37 2010
**************************************************
root-server03:/etc>

Thanks again to all who've responded
Gary Lanzi

[Bill Hassell]

+ export ORACLE_HOME PATH
+ [ ! ORA_DB = ORA_DB ]
+ /u/app/oracle/local/bin/oracle.startup /u/app/oracle/local
+ exit
+ exit

set -x traces the execution of the program. Apparently, all the script does is to run oracle.startup. If you read the start/stop script, you'll see the above code. Now you'll have to move the set -x option to the oracle.startup file (assuming it is also a script).

[Wim Rombauts]

Hello Gary,

I am not familiar with 11g yet, but at 10g, Oracle was starting some dummy clusterware process first (CSS daemon at 10g R2), which was absolutley necessary to be there to allow the Oracle database to start up.
Do you have anything like /sbin/init.d/init* ?

I remember in our case that this daemon didn't start up fine, so that the Oracle database itself refused to start.

[Dennis Handly]

>Wim: so that the Oracle database itself refused to start.

Was this logged in some obvious place?