Oracle DB will not start on alt node in cluster.

[Gino Castoldi_2]

Hi,

Server:
HP-UX 11.11 MC/SG 11.15 Oracle 8.1.7.0 OVO 7.18 NNM 6.2

Currently OVO and the Oracle database are running fine on the primary node in a two node cluster. Whenever we attempt to fail-over
to the alternate node it fails.

log file output listed below:

04/14/04 05:56:23 ERROR ovoareqsdr (Request Sender)(8779) [dcesec.c:925]: Can't bind to DCE security service: No currently established network identi)
Database: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
HP-UX Error: 2: No such file or directory
(OpC50-15)
Could not connect to database ov_net.
Please look if the database processes are running. (OpC50-2)
Connect to database failed. (OpC40-410)

Any ideas as to what could be wrong?
Not sure if it's really an Oracle issue or
a system type problem (kernel/shared memory?).

This used to work fine about two months ago
(the last time we failed over).
No real changes have been made.

10 points to any good answer.
TIA, Gino

[Massimo Bianchi · Accepted solution]

Hi GIno,
i would check the following things:

- did you installed any dce patches, like the ones for the worm of november ?

- when you do the switch, does oracle start or not ? You can check with a simple "ps -ef | grep ora ". Check in the oracle alert_openview.log to see if there is any outstanding error.

- are /etc/listener.ora and /etc/tnsnames.ora, if present, equal ?

- is there any difference in patching between the two system ?

- is there any password expiry policy, that cloud have prevented the startup of oracle ?

Best regards,
Massimo

[Gino Castoldi_2]

Hi,

- did you installed any dce patches, like the ones for the worm of november ?
No.

- when you do the switch, does oracle start or not ? No, Oracle does not start.

- Check in the oracle alert_openview.log to see if there is any outstanding error.
I checked the alert_openview.log and listener.log and there's not errors.

- are /etc/listener.ora and /etc/tnsnames.ora, if present, equal ?
They only exist on the shared disks which are only present on whatever node is running
the OVO package.

- is there any difference in patching between the two system ?
No.

- is there any password expiry policy, that cloud have prevented the startup of oracle ?
No.

I am beginning to think it's a resource issue
so I'm planning to reboot the server early tomorrow morning.

10 points to any good answer.
TIA, Gino

[G. Vrijhoeven]

Hi,

Is your listerner running?
# su - oracle -c "lsnrctl status" ( start to start)
If it is running, is it set up ok, package ipadress, not server ipadress is tnsnames.ora?

HTH,

Gideon

[Gino Castoldi_2]

Hi,

>Is your listener running?
Yes.

># su - oracle -c "lsnrctl status" ( start >to start)
>If it is running, is it set up ok, package >ipadress, not server ipadress is >tnsnames.ora?
The setup is ok. The virtual node's ip address is in the tnsnames.ora file.

10 points to any good answer.
TIA, Gino

[Ashwani Kashyap]

Is dce running on the failover node .
Try stopping and restarting dce , and then failover .

On the second node , there might be a possiblity that oracle didnot release some of the semaphores . Check with ipcs and remove those semaphores and then try to failover .

[Fred Ruffet]

I don't think it's a listener problem : The message indicates Oracle is down (message for listener down is different).

Can you start Oracle manually ?
If you start Oracle manually and then fail-over, is it working ? In that case, it would be the fail-over script that doesn't start Oracle : its log should help. (You won't have any log in alert.log, as long as Oracle isn't started.)

[Gino Castoldi_2]

Hi,

>Is dce running on the failover node .
Yes.
>Try stopping and restarting dce , and then failover .
I will reboot the server tomorrow.

>On the second node , there might be a possiblity that oracle did not release some of the semaphores . Check with ipcs and remove those semaphores and then try to failover .
A reboot will clear all semaphores, shared memory, etc. This is a production server so I have to wait until tomorrow.

10 points to any good answer.
TIA, Gino

[Gino Castoldi_2]

Hi,

>Can you start Oracle manually ?
If you start Oracle manually and then fail-over, is it working ? In that case, it would be the fail-over script that doesn't start Oracle : its log should help. (You won't have any log in alert.log, as long as Oracle isn't started.)

After I reboot the server I'll start the package(OVO) and Oracle by switching it over to that node. If that doesn't work then I'll try to manually start the Oracle database.

10 points to any good answer.
TIA, Gino

[Massimo Bianchi]

Hi Gino,
you said that oracle didn't start, and that you think that it's a resource issue.

Then, what are the errors in the alert_openview.log ?

Regards,
Massimo

[Gino Castoldi_2]

Hi,

>you said that oracle didn't start, and that you think that it's a resource issue.
Then, what are the errors in the alert_openview.log ?

There are no errors in the alert_openview.log & listener.log log files.

10 points to any good answer.
TIA, Gino

[Massimo Bianchi]

No errors ?

IS there, at least, the classic lines from the startup of the instance ?

For the resource issue: please issue, on the alternate node, the following commands

ipcs -ma

and

ipcs -s

and see if there is any shared segment/ semaphore left there.

If we cannot confirm that the problemis memory, your reboot may result useless.

Regards,
Massimo

[Gino Castoldi_2]

Hi,

This morning I rebooted the alternate server in the cluster and when I switched the package (OVO/Oracle) onto that node it worked fine. A reboot was necessary to clear up whatever resource issues there were.

Thank you again for your help.
Gino