Oracle RAC 10gR2 ONS daemon down frequently

[Andy_112]

Hi,

We had installed a 10gR2 RAC Clusterware on a HP-UX 11.11 2 nodes cluster system. Starting from beginning all services are running. But around 40 mins later the ONS reported OFFLINE but the unix processes are still there.

We tried to use crsctl stop crs to shutdown the clusterware. All CRS processes can be shutdown normally except ons. What we can do is to kill it by -9.

Without a healthy CRS we cannot move to next step to create/upgrade any cluster database on 10gR2.

We had logged the call to oracle already but still could not find out the root cause. Any idea or experience you can share? Thanks in advance.

Andy

[Andy_112]

Sorry. Additional info can be provided as follow:

2008-05-08 21:44:45.708: [ RACG][1] [17533][1][ora.hkdev06.ons]: 0]
{node = hkdev06, port = 6200}
Adding remote host hkdev06:6200
onscfg[1]
{node = hkdev01, port = 6200}
Adding remote host hkdev01:6200
onsctl: ons started

2008-05-08 22:44:49.857: [ RACG][1] [22659][1][ora.hkdev06.ons]: clsrcgetprsr
ctx: prsr_init_ext returned rc = 3

2008-05-08 22:44:50.199: [ RACG][1] [22659][1][ora.hkdev06.ons]: clsrons_init
failed, stat = 504, ocrerr = 32
ons is not running ...

Same log error can be found from both nodes.

Thx again
Andy

[Steven E. Protter]

Shalom Andy,

The errors in your second post seem to indicate that some portion of the setup requirements have not been met.

You should first insure that no setup steps have been skipped.

It is very important to meet any OS patch requirements listed.

If you are sure, then most likely you need to contact Oracle support for assistance in what oracle patches need to be installed.

SEP

[Andy_112]

Hi Stephen,

Yes we had logged a call to Oracle Support for a months and finally they classified that it "may" be a bug and they had already referred to lab for further investigation. So I just want to find my way from other direction.
The installation was smooth and zero error were reported. Anyway let me re-verify the OS patch again. Thanks for your suggestion.
Andy

[Andy_112]

Hi,

I checked all patch level and quite sure it is qualified with Oracle requirement. But still got no luck to fix the problem. May be I can provide more information if anyone can give me more hints to solve this.

When I used crsctl to stop all CRS services, suppose all CRS daemon should be shutdown properly. But our case is that:

$ crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
$ onsctl ping
clsrons_init failed, stat = 504, ocrerr = 32
ons is not running ...
$ ps -ef|grep ons|grep oracle
oracle 24272 24271 0 14:02:18 ? 16:30 /oracle/product/CRS/opmn/bin/ons -d
oracle 24271 1 0 14:02:18 ? 0:00 /oracle/product/CRS/opmn/bin/ons -d

The daemon is still running but just got no responese. In this case I can use kill -9 to kill all ons and restart the whole CRS to enjoy my 30-40 mins CRS environment.

Any idea? Thanks in advance
Andy