Failed to Start oracle instance

[Jacky Wing]

I am having a problem with SG integration with Oracle. I am using HP-UX 11i v3, MC/SG A.11.19.00 & ECMT B.05.00.

I've setup the cluster config, checked it & applied with the package in question, when running the cluster, nodes run normally, however package halts and I get the "Failed to start oracle instance".

As far as I know, I set up the oracle params in the package conf file; and I add the command: " $SGCONF/scripts/ecmt/oracle/tkit_module.sh start" to the start function.
the external script calls tkit_module.sh with command start which calls toolkit.sh which calls haoracle.sh which calls haoracle_sql.sh with command start.

Apparently all is going fine, and I get the "Database Mounted"
"Database Opened"
Then I get: "Oracle instance startup failed"
then: " ERROR: Function oracle_run_cmds: Failed to start Oracle instance"
and then the scripts kills oracle instance and the package fails.

I tried to work around the haoracle_sql.sh script to know what is happening with no success, added some echos, ... with no success

I have no DBA skills, & I barely understand the info written in haoracle_sql.sh

Anybody has any idea?

[melvyn burnard]

you need to have someone who knows Oracle to make sure you have all the correct variables set in the toolkit.
Highly possible you are not starting the database correctly

[Jacky Wing]

the DB seems to start from the logs, but it is reported as not started and is sent the Kill command from the toolkit script.

the DBA over here says his db is running with no problem, so m left with solving this issue form my side.

If i try to run the db without the listener & without the other variables it runs normally as per the dba. I don't see how it's failing.

do you have any steps I can troubleshoot the problem with? I tried messing up the haoracle_sql.sh script and the other scripts also without any success.

I think it's stupid to see if the error lies in the ecmt scripts but I'm only left with this possibility.

Any help is appreciated!!

[Shibin_2]

I understand that your problem arises, when your DB starts while the package start-up.

Do you have any hint from package control logs?

You also mentioned, manual DB startup is success. Isn't because of user? Oracle DB's should start using their own admin user id, not root user. Have you checked that option in the script ?

[Jacky Wing]

from the logs, nothing seems clear, just that the listener started, the "DB mounted" "DB Opened"; then "Failed to start Oracle" and the script kills the oracle and the package halts.

I tried to play with the haoracle_sql.sh script to get an idea of what is happening, & I removed a line where the status_return=1 ; the db started and the package started for seconds than it was killed by the monitoring app & the listener monitoring application. I guess it wasn't started after all.

I want to be able to do the commands written in the script manually but it's not easy with all the characters and all the if's.

The oracle user is defined in the: ecmt/oracle/oracle/ORACLE_ADMIN as oracle so this one is clear.

[Jacky Wing]

No ideas?

[Horia Chirculescu]

Hello,

Double check the variables from haoracle.conf file.

Read the documentation from:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02496200/c02496200.pdf

Best regards,
Horia.

[Horia Chirculescu]

>I tried to play with the haoracle_sql.sh script to get an idea of what is happening, & I removed a line where the status_return=1 ; the db started and the package started for seconds than it was killed by the monitoring app & the listener monitoring application. I guess it wasn't started after all.

You should restore the haoracle_sql.sh to the original version.

Best regards
Horia.

[Jacky Wing]

I did restored it after that,
I restarted the cluster configuration from scratch, I even deleted the configuration and re-created it from zero.
I am still faced with the same problem, I read all the ECMT docs & the oracle integration.
I double checked all the params, & the whole procedure. Nothing :S,

I am re-creating the cluster now,

[Horia Chirculescu]

Are you able to manually start your instance from sqlplus (check the document - page 18) ?


Best regards,
Horia.

[Jacky Wing]

I am able to start it from command line.

[Jacky Wing]

Configured it using legacy packages --> works fine.

Configured it using serviceguard manager, same error :S.

A Stupid question right now, Modular packages & ... is supported with Oracle 9i?

[Shibin_2]

From this document,

http://www.filibeto.org/unix/hp-ux/lib/cluster/serviceguard/ecmt/relnotes-ECMT-05.00-T1909-90066.pdf

The Enterprise Cluster Master Toolkit version B.05.00 supports the following:
â ¢ Oracle 9i
â ¢ Oracle 9i RAC
â ¢ Oracle 10g R1
â ¢ Oracle 10g R2 Database Server with Automatic Storage Management
â ¢ Oracle 11gR1 Database Server
â ¢ Oracle 11g R1 Database Server with Automatic Storage Management on the Itanium servers

It is mentioning some patches PHSS_38424 . Do you have it?

Another security bulletin.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01894850

[Horia Chirculescu]

In order to find out what is wrong, you should disable monitoring.

Please read the pdf file from above to find out how to do that (basically you only need to create a file with .debug extension - pag. 48).

If the instance will fail to start, you have a problem with the configuration file.

Best regards
Horia.

[Jacky Wing]

Installed the patch in question, I also ran the patch inventory & all the recommended patches are there.
I tried the legacy package with the new scripts --> doesn't work.
I tried the modular package with B.03 ECMT scripts "Surprisingly works"

I am going for Legacy with ECMT B.03. I guess there is some kind of incompatibility issue.

[Terry.giblin]

"Failed to start oracle instance"

I would ask your DBA's to check and kill all oracle zombie processes, before starting oracle just in case.

If that does not work ask your DBA's to check who is starting the instance?

Then check the errlog file and directory file size and type is very important. I hope you have plenty of space...

[Terry.giblin]

Dear Jacky,

I am going off memory here and I am not a DBA, but have you considered ipcs - 'interprocess communications facilities' and shared memory segments?

Your Oracle Database may not start if MC/SG does not shutdown correctly because there is not enough shared memory segments.

Its worth checking you need to write a short script to clean out any old ipcs entries.

Ask a DBA!

[Jacky Wing]

Well, the DB starts from command line when I just run the package to mount fs.
what i found strange is that the logs show:
- DB mounted
- DB opened
then the scripts logs, DB startup failed. how come it failed if the DB is opened;
Oracle logs show that the DB starts, then it is sent the kill signal.

I tried to log more info but I don't know why the logs didn't show any more info.

legacy with B.03 works for me, why should I use the new scripts?. I spent a great deal of time trying to make it work, if it doesn't then ll just move to the old working scripts.

I guess there is something wrong in the new scripts, compatibility issue or smthg.