Service Cluster fail over node1 to node2,,,

[Kannandgl_1]

Dear Friends ,

Hope your are helping me.We have 2 node Service guard cluster running on HP unix 11.23
Before 20 days i am facing same problem fail over at the time some file mismatch in our listener and tns entry on my DB side after modified there in no fail over.After 20 days 2 day i am getting fail over node 1 to node 2.the package is automatically disabled on node1.
# cmviewcl -v

CLUSTER STATUS
cluster1 up

NODE STATUS STATE
hogisdata1 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/3/1/0 lan0
PRIMARY up 0/4/2/0 lan2

NODE STATUS STATE
hogisdata2 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/3/1/0 lan0
PRIMARY up 0/4/2/0 lan2

PACKAGE STATUS STATE AUTO_RUN NODE
orapkg up running enabled hogisdata2

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 0 0 orapkg
Subnet up 10.38.1.0
Subnet up 10.10.10.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up disabled hogisdata1
Alternate up enabled hogisdata2 (current)



Error from oracle controller.log:
===================================
kill: 3457: no such process
Process failed: 3457 - Process-type: 0 }
Starting Oracle Server Listener...

LSNRCTL for HPUX: Version 10.2.0.3.0 - Production on 12-JUN-2010 06:33:26

Copyright (c) 1991, 2006, Oracle. All rights reserved.

TNS-01106: Listener using listener name LISTENER has already been started
Oracle Server Listener start failed.
Starting Oracle Server Listener...

LSNRCTL for HPUX: Version 10.2.0.3.0 - Production on 12-JUN-2010 06:33:26

Copyright (c) 1991, 2006, Oracle. All rights reserved.

TNS-01106: Listener using listener name LISTENER has already been started
Oracle Server Listener start failed.
Starting Oracle Server Listener...

LSNRCTL for HPUX: Version 10.2.0.3.0 - Production on 12-JUN-2010 06:33:26

Copyright (c) 1991, 2006, Oracle. All rights reserved.

TNS-01106: Listener using listener name LISTENER has already been started
Oracle Server Listener start failed.

=======================================
root 6804 1 12 Jun 10 ? 2:27 /opt/hpsmc/avc/bin/monitorsvcd
oracle 3351 1 0 06:32:20 ? 0:00 oracleewagdb (LOCAL=NO)
Stopping EM.....

SQL*Plus: Release 10.2.0.3.0 - Production on Sat Jun 12 06:33:27 2010

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> ORACLE instance shut down.
SQL> Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
Oracle abort done.
Stopping Oracle Server Listener...

LSNRCTL for HPUX: Version 10.2.0.3.0 - Production on 12-JUN-2010 06:33:29

Copyright (c) 1991, 2006, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cluster1)(PORT=1521)))
The command completed successfully
Oracle Server Listener stop done.

*** tnslsnr process has stopped. ***
==============

+ date
+ print \n\t########### Node "hogisdata1": Halting package at Sat Jun 12 06:33:31 WAT 2010 ###########

########### Node "hogisdata1": Halting package at Sat Jun 12 06:33:31 WAT 2010 ###########
+ stop_resources
+ halt_services
Jun 12 06:33:31 - Node "hogisdata1": Halting service orapkg
cmhaltserv : Service name orapkg is not running.
+ customer_defined_halt_cmds
hogisdata1
Sat Jun 12 06:33:31 WAT 2010
.:/orabin/oracle/product/10g/db/bin:/sbin:/tools/bin:/usr/sbin:/usr/sbin:/usr/bin:/usr/sbin:/etc:/bin:/orabin/oracle/product/10g/db/audit/scripts:/opt/clic/bi
n://orabin/oracle/product/10g/db/jdk/bin:/orabin/oracle/product/10g/db/dcm/bin:/orabin/oracle/product/10g/db/opmn/bin:/orabin/oracle/product/10g/db/Apache/Apa
che/bin:/opt/java1.4/bin:/orabin/oracle/product/10g/db/bin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/etc:/bin

*** /etc/cmcluster/orapkg/orapkg.sh called with stop argument. ***


"hogisdata1": Shutting down Oracle SESSION ewagdb at Sat Jun 12 06:33:31 WAT 2010
Stopping OPMN managed processes...
Stopping EM.....
Stopping Oracle Server .....

SQL*Plus: Release 10.2.0.3.0 - Production on Sat Jun 12 06:33:31 2010

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Connected to an idle instance.
=======================

Regards
Rajamani.............

[Basheer_2 · Accepted solution]

Hello,

you have 2 nodes, and both are up and running.

NODE STATUS STATE
hogisdata1 up running

NODE STATUS STATE
hogisdata2 up running


orapkg is running in hogisdata2
PACKAGE STATUS STATE AUTO_RUN NODE
orapkg up running enabled hogisdata2

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual <--- this means if you want to run this in node1, you have to do it manually.

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up disabled hogisdata1
Alternate up enabled hogisdata2 (current)

I dont see anything wrong here. SG is behainvg OK.

if you want to run orapkg in node1, run
cmmodpkg commands.

[S. Ney]

Not sure if this is what you are asking however the oracle listner should be set to the floating IP for hostname so that is fails back and forth between the servers. Have your dba's verify that the
listener.ora file HOST = (floating IP address)

Was your package successfully working before?
Who made changes? SA's to service guard or dba's to oracle?

If your setup is active/passive oracle should only run on one server at a time. Your package scripts may or may not contain the shutdown commands for oracle processes. Our application uses custom package scripts but yours may have something similar:
# a shutdown abort then a SIGKILL (if needed) to each of the monitored processes.
${0} kill $PACKAGE_NAME $P_PREFIX &
# Stop the listener if needed
if [[ ${STARTUP_LISTENER} = yes ]]
then
su oracle -c "${ORACLE_HOME}/bin/lsnrctl stop lsn_${ORACLE_SID}"
# Shutdown Oracle with an immediate argument
su - oracle <export ORACLE_SID=#####
$ORACLE_HOME/bin/sqlplus '/nolog'
connect / as sysdba
shutdown immediate
exit

If the SA's did any package or cluster work was cmcheckconf on package&/or cluster done?