Oracle process ora_pmon_?????? has failed

[Eudo Jimenez_2]

Hello All,

Current State:
Listener and database starts manually without any error.I can connect to database and make query.
Listener and database stops manually without any error.
I use HP Enterprise Cluster Master Toolkit (ECMT) version B.03.00 as start/stop/monitor scripts
Cmruntpkg can start the database but report an error. Here is a result of cmrunpkg:


########### Node "HP1NP0": Starting package at Mon Sep 4 11:14:26 SAT 2006 ###########
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg01 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg02 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg03 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg04 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg03" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg04" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg05 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg06 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg07 with exclusive option.
Sep 4 11:14:26 - Node "HP1NP0": Activating volume group /dev/vg08 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg07" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg08" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg06" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg05" has been successfully changed.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg09 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg10 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg11 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg12 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg11" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg10" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg09" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg12" has been successfully changed.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg13 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg14 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg15 with exclusive option.
Sep 4 11:14:27 - Node "HP1NP0": Activating volume group /dev/vg16 with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg15" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg13" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg14" has been successfully changed.
Activated volume group in Exclusive Mode.
Volume group "/dev/vg16" has been successfully changed.
Sep 4 11:14:28 - Node "HP1NP0": Checking filesystems:
/dev/vg01/lvol1
/dev/vg02/lvol2
/dev/vg03/lvol3
/dev/vg04/lvol4
/dev/vg05/lvol5
/dev/vg06/lvol6
/dev/vg07/lvol7
/dev/vg08/lvol8
/dev/vg09/lvol9
/dev/vg10/lvol10
/dev/vg11/lvol11
/dev/vg12/lvol12
/dev/vg13/lvol13
/dev/vg14/lvol14
/dev/vg15/lvol15
/dev/vg16/lvol16
/dev/vg01/rlvol1:file system is clean - log replay is not required
/dev/vg03/rlvol3:file system is clean - log replay is not required
/dev/vg04/rlvol4:file system is clean - log replay is not required
/dev/vg02/rlvol2:file system is clean - log replay is not required
/dev/vg07/rlvol7:file system is clean - log replay is not required
/dev/vg06/rlvol6:file system is clean - log replay is not required
/dev/vg05/rlvol5:file system is clean - log replay is not required
/dev/vg08/rlvol8:file system is clean - log replay is not required
/dev/vg09/rlvol9:file system is clean - log replay is not required
/dev/vg11/rlvol11:file system is clean - log replay is not required
/dev/vg12/rlvol12:file system is clean - log replay is not required
/dev/vg10/rlvol10:file system is clean - log replay is not required
/dev/vg14/rlvol14:file system is clean - log replay is not required
/dev/vg16/rlvol16:file system is clean - log replay is not required
/dev/vg13/rlvol13:file system is clean - log replay is not required
/dev/vg15/rlvol15:file system is clean - log replay is not required
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg01/lvol1 at /lvolora1
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg02/lvol2 at /lvolora2
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg03/lvol3 at /lvolora3
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg04/lvol4 at /lvolora4
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg05/lvol5 at /lvolora5
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg06/lvol6 at /lvolora6
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg07/lvol7 at /lvolora7
Sep 4 11:14:29 - Node "HP1NP0": Mounting /dev/vg08/lvol8 at /lvolora8
Sep 4 11:14:31 - Node "HP1NP0": Mounting /dev/vg09/lvol9 at /lvolora9
Sep 4 11:14:32 - Node "HP1NP0": Mounting /dev/vg10/lvol10 at /lvolora10
Sep 4 11:14:32 - Node "HP1NP0": Mounting /dev/vg11/lvol11 at /lvolora11
Sep 4 11:14:32 - Node "HP1NP0": Mounting /dev/vg12/lvol12 at /lvolora12
Sep 4 11:14:33 - Node "HP1NP0": Mounting /dev/vg13/lvol13 at /lvolora13
Sep 4 11:14:34 - Node "HP1NP0": Mounting /dev/vg14/lvol14 at /lvolora14
Sep 4 11:14:35 - Node "HP1NP0": Mounting /dev/vg15/lvol15 at /lvolora15
Sep 4 11:14:35 - Node "HP1NP0": Mounting /dev/vg16/lvol16 at /lvolora16
Sep 4 11:14:37 - Node "HP1NP0": Adding IP address 10.1.18.201 to subnet 10.1.16.0
Sep 4 11:14:37 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/toolkit.sh called with start argument. ***
Sep 4 11:14:37 - Node "HP1NP0" : Starting Oracle SESSION STUDUM at Mon Sep 4 11:14:37 SAT 2006
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2003 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2003 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.


RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.


Hewlett-Packard Company
3000 Hanover Street
Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).
Not a terminal
stty: : Not a typewriter
stty: : Not a typewriter


2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

stty: : Not a typewriter

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 04-SEP-2006 11:14:56

Copyright (c) 1991, 2002, Oracle Corporation. All rights reserved.

Starting /lvolora12/app/product/9.2.0.6/bin/tnslsnr: please wait...

TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
System parameter file is /lvolora12/app/product/9.2.0.6/network/admin/listener.ora
Log messages written to /lvolora12/app/product/9.2.0.6/network/log/hp1n0_listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.18.201)(PORT=1555)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.18.201)(PORT=2481))(PROTOCOL_STACK=(PRESENTATION=GIOP)(SESSION=RAW)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
STATUS of the LISTENER
------------------------
Alias HP1N0_LISTENER
Version TNSLSNR for HPUX: Version 9.2.0.6.0 - Production
Start Date 04-SEP-2006 11:14:57
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security OFF
SNMP OFF
Listener Parameter File /lvolora12/app/product/9.2.0.6/network/admin/listener.ora
Listener Log File /lvolora12/app/product/9.2.0.6/network/log/hp1n0_listener.log
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.18.201)(PORT=1555)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.1.18.201)(PORT=2481))(PROTOCOL_STACK=(PRESENTATION=GIOP)(SESSION=RAW)))
Services Summary...
Service "BOREPO.world" has 1 instance(s).
Instance "BOREPO", status UNKNOWN, has 1 handler(s) for this service...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "SERVDESK.world" has 1 instance(s).
Instance "SERVDESK", status UNKNOWN, has 1 handler(s) for this service...
Service "STUDUM.world" has 1 instance(s).
Instance "STUDUM", status UNKNOWN, has 1 handler(s) for this service...
Service "STUDUT.world" has 1 instance(s).
Instance "STUDUT", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
logout
Sep 4 11:14:57 - Node "HP1NP0" Oracle startup done.
Sep 4 11:14:57 - Node "HP1NP0": Starting service ORA_MONITOR_STUDUM using
"/etc/cmcluster/pkg_ora_suagm1/studum/haoracle.sh monitor"
Sep 4 11:14:57 - Node "HP1NP0": Starting service ORA_MONITOR_STUDUT using
"/etc/cmcluster/pkg_ora_suagm1/studut/haoracle.sh monitor"
Sep 4 11:14:57 - Node "HP1NP0": Starting service ORA_MONITOR_SERVDESK using
"/etc/cmcluster/pkg_ora_suagm1/servdesk/haoracle.sh monitor"
Sep 4 11:14:57 - Node "HP1NP0": Starting service ORA_MONITOR_BOREPO using
"/etc/cmcluster/pkg_ora_suagm1/borepo/haoracle.sh monitor"
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/haoracle.sh called with monitor argument. ***

########### Node "HP1NP0": Package start completed at Mon Sep 4 11:14:57 SAT 2006 ###########
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studut/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/servdesk/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/borepo/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_lgwr_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_reco_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_lgwr_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_reco_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_lgwr_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_lgwr_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_reco_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_reco_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studut/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/servdesk/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/borepo/haoracle.sh called with monitor argument. ***
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_pmon_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_STUDUT has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_dbw0_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_SERVDESK has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_STUDUM has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_ckpt_BOREPO has failed
Sep 4 11:14:57 - Node "HP1NP0" Oracle process ora_smon_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/haoracle.sh called with monitor argument. ***
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studut/haoracle.sh called with monitor argument. ***
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/servdesk/haoracle.sh called with monitor argument. ***
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/borepo/haoracle.sh called with monitor argument. ***
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_pmon_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_pmon_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_dbw0_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_pmon_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_pmon_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_dbw0_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_dbw0_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_dbw0_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_ckpt_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_ckpt_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_ckpt_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_ckpt_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_smon_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_STUDUM has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_lgwr_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_STUDUT has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_BOREPO has failed
Sep 4 11:14:58 - Node "HP1NP0" Oracle process ora_reco_SERVDESK has failed
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.
Sep 4 11:14:58 - Node "HP1NP0" Exiting Toolkit Monitoring because one of the key oracle process has failed.

########### Node "HP1NP0": Halting package at Mon Sep 4 11:14:58 SAT 2006 ###########
Sep 4 11:14:58 - Node "HP1NP0": Halting service ORA_MONITOR_STUDUM
cmhaltserv: Service name ORA_MONITOR_STUDUM is not running.
Sep 4 11:14:58 - Node "HP1NP0": Halting service ORA_MONITOR_STUDUT
cmhaltserv: Service name ORA_MONITOR_STUDUT is not running.
Sep 4 11:14:58 - Node "HP1NP0": Halting service ORA_MONITOR_SERVDESK
cmhaltserv: Service name ORA_MONITOR_SERVDESK is not running.
Sep 4 11:14:58 - Node "HP1NP0": Halting service ORA_MONITOR_BOREPO
cmhaltserv: Service name ORA_MONITOR_BOREPO is not running.
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/toolkit.sh called with stop argument. ***
Sep 4 11:14:58 - Node "HP1NP0" : Aborting Oracle SESSION STUDUM at Mon Sep 4 11:14:58 SAT 2006
Sep 4 11:14:58 - Node "HP1NP0" *** /etc/cmcluster/pkg_ora_suagm1/studum/toolkit.sh called with kill argument. ***
ttytype: couldn't open /dev/tty for reading
stty: : Not a typewriter
(c)Copyright 1983-2003 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2003 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.


RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.


Hewlett-Packard Company
3000 Hanover Street
Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).
Not a terminal
stty: : Not a typewriter
stty: : Not a typewriter


2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

stty: : Not a typewriter

SQL*Plus: Release 9.2.0.6.0 - Production on Mon Sep 4 11:14:58 2006

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.


Connected to:
Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production

SQL> ORACLE instance shut down.
SQL> Disconnected from Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 - Production

LSNRCTL for HPUX: Version 9.2.0.6.0 - Production on 04-SEP-2006 11:14:58

Copyright (c) 1991, 2002, Oracle Corporation. All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> LSNRCTL> Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
The command completed successfully
LSNRCTL> logout
Sep 4 11:14:58 - Node "HP1NP0" *** ora_pmon_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" *** ora_dbw0_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" *** ora_ckpt_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" *** ora_smon_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" *** ora_lgwr_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" *** ora_reco_STUDUM process is not running. ***
Sep 4 11:14:58 - Node "HP1NP0" Killing the kill_hung_processes script

Sep 4 11:14:58 - Node "HP1NP0": Remove IP address 10.1.18.201 from subnet 10.1.16.0
Sep 4 11:14:59 - Node "HP1NP0": Unmounting filesystem on /dev/vg16/lvol16
Sep 4 11:14:59 - Node "HP1NP0": Unmounting filesystem on /dev/vg15/lvol15
Sep 4 11:14:59 - Node "HP1NP0": Unmounting filesystem on /dev/vg14/lvol14
Sep 4 11:14:59 - Node "HP1NP0": Unmounting filesystem on /dev/vg13/lvol13
Sep 4 11:15:00 - Node "HP1NP0": Unmounting filesystem on /dev/vg12/lvol12
Sep 4 11:15:00 - Node "HP1NP0": Unmounting filesystem on /dev/vg11/lvol11
Sep 4 11:15:01 - Node "HP1NP0": Unmounting filesystem on /dev/vg10/lvol10
Sep 4 11:15:01 - Node "HP1NP0": Unmounting filesystem on /dev/vg09/lvol9
Sep 4 11:15:02 - Node "HP1NP0": Unmounting filesystem on /dev/vg08/lvol8
Sep 4 11:15:02 - Node "HP1NP0": Unmounting filesystem on /dev/vg07/lvol7
Sep 4 11:15:02 - Node "HP1NP0": Unmounting filesystem on /dev/vg06/lvol6
Sep 4 11:15:02 - Node "HP1NP0": Unmounting filesystem on /dev/vg05/lvol5
Sep 4 11:15:03 - Node "HP1NP0": Unmounting filesystem on /dev/vg04/lvol4
Sep 4 11:15:03 - Node "HP1NP0": Unmounting filesystem on /dev/vg03/lvol3
Sep 4 11:15:03 - Node "HP1NP0": Unmounting filesystem on /dev/vg02/lvol2
Sep 4 11:15:03 - Node "HP1NP0": Unmounting filesystem on /dev/vg01/lvol1
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg01
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg02
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg01" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg03
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg02" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg04
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg03" has been successfully changed.
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg04" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg05
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg06
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg05" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg07
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg06" has been successfully changed.
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg07" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg08
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg08" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg09
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg10
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg09" has been successfully changed.
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg10" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg11
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg12
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg11" has been successfully changed.
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg12" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg13
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg14
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg13" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg15
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg14" has been successfully changed.
Sep 4 11:15:05 - Node "HP1NP0": Deactivating volume group /dev/vg16
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg15" has been successfully changed.
Deactivated volume group in Exclusive Mode.
Volume group "/dev/vg16" has been successfully changed.

########### Node "HP1NP0": Package halt completed at Mon Sep 4 11:15:05 SAT 2006 ###########



WHEN I SET MANTENANCE MODE (touch /etc/cmcluster/pkg_ora_suagm1/{SID_NAME}/oracle.debug to stop the monitor) THE PACKAGE RUN OK

I saw that there are a patch to Oracle Toolkit B.02.21 reference to this situations, but I am ussing B.03.00

Attached the pkg.conf, pkg.sh, haoracle.mom and haoracle.sh

[Eudo Jimenez_2]

I am checking on haoracle.mon and the command line
act_procs=`UNIX95=YES ps -u ${ORACLE_ADMIN} -o args | grep -v grep | grep -c $grepString`
return act_procs=0 and this not thru, there are many process ora_pmon_${SID_NAME}, ora_dbw0_${SID_NAME}, ora_smon_${SID_NAME}, etc.

HP1NP0 #ps -ef | grep oraclust
oraclust 7774 1 0 12:04:25 ? 0:00 ora_d002_BOREPO
oraclust 7444 1 0 12:03:45 ? 0:00 ora_dbw0_STUDUM
oraclust 7500 1 0 12:03:48 ? 0:00 ora_d017_STUDUM
oraclust 7770 1 0 12:04:25 ? 0:00 ora_d000_BOREPO
oraclust 7448 1 0 12:03:46 ? 0:00 ora_lgwr_STUDUM
oraclust 7492 1 0 12:03:47 ? 0:00 ora_d013_STUDUM
oraclust 7575 1 0 12:03:58 ? 0:00 ora_d000_STUDUT
oraclust 7633 1 0 12:03:59 ? 0:00 ora_d017_STUDUT
oraclust 7454 1 0 12:03:46 ? 0:00 ora_reco_STUDUM
oraclust 7558 1 0 12:03:58 ? 0:00 ora_ckpt_STUDUT
oraclust 7587 1 0 12:03:59 ? 0:00 ora_d006_STUDUT
oraclust 7766 1 0 12:04:25 ? 0:00 ora_s000_BOREPO
oraclust 7573 1 0 12:03:58 ? 0:00 ora_s004_STUDUT
oraclust 7472 1 0 12:03:47 ? 0:00 ora_d003_STUDUM
oraclust 7709 1 0 12:04:15 ? 0:00 ora_d003_SERVDESK
oraclust 7707 1 0 12:04:15 ? 0:00 ora_d002_SERVDESK
oraclust 7598 1 0 12:03:59 ? 0:00 ora_d010_STUDUT
oraclust 7756 1 0 12:04:25 ? 0:00 ora_dbw1_BOREPO
oraclust 7620 1 0 12:03:59 ? 0:00 ora_d012_STUDUT
oraclust 7482 1 0 12:03:47 ? 0:00 ora_d008_STUDUM
oraclust 7583 1 0 12:03:59 ? 0:00 ora_d004_STUDUT
oraclust 7508 1 0 12:03:48 ? 0:00 ora_arc1_STUDUM
oraclust 7699 1 0 12:04:15 ? 0:00 ora_s000_SERVDESK
oraclust 7478 1 0 12:03:47 ? 0:00 ora_d006_STUDUM
oraclust 7476 1 0 12:03:47 ? 0:00 ora_d005_STUDUM
oraclust 7484 1 0 12:03:47 ? 0:00 ora_d009_STUDUM
oraclust 7693 1 0 12:04:15 ? 0:00 ora_ckpt_SERVDESK
oraclust 7705 1 0 12:04:15 ? 0:00 ora_d001_SERVDESK

[Steven E. Protter]

Shalom,

1) Run a connect test with sqlplus and see if your database is up or down. It might be working wright.

2) If its not working correctly check permissions/ownership and the oracle alert logs and finall the relavent section of the cluster logs versus the entire verbose output.

SEP

[Eudo Jimenez_2]

The databases are OK.

I thing that the monitor script (haoracle.mom) had problems

[Eudo Jimenez_2]

I am checking the haoracle.mon script and do not understand de command lines

grepString=`echo "${MONITOR_PROCESSES[*]}"|sed 's/ /$ \-e ^/g'`
grepString="-e ^$grepString$"


On my case, when i do this command line in prompt and do echo $grepString, it return
-e ^ora_pmon_STUDUM$ -e ^ora_dbw0_STUDUM$ -e ^ora_ckpt_STUDUM$ -e ^ora_smon_STUDUM$ -e ^ora_lgwr_STUDUM$ -e ^ora_reco_STUDUM$
when I did echo $act_procs after do the command line
act_procs=`UNIX95=YES ps -u ${ORACLE_ADMIN} -o args | grep -v grep | grep -c $grepString`
It return
act_procs=0 (This is wrong)

I was not sure about the ^ and $ caracter use and I changed the sintaxys in order to test.

grepString=`echo "${MONITOR_PROCESSES[*]}"|sed 's/ / \-e /g'`
grepString="-e $grepString"
when I did echo $act_procs after do the command line
act_procs=`UNIX95=YES ps -u ${ORACLE_ADMIN} -o args | grep -v grep | grep -c $grepString`
It return
act_procs=6. This is OK

NOTE: I did all these commands in interactive mode setting all needed variable.

[Carsten Krege]

The character "^" stands for the beginning of the line. The "$" stands for the end of the line. The expression "grep -e ^foobar$" therefore finds lines that only contain the string "foobar" without any leading or trailing characters (like spaces). See also the regexp(5) manual page.

In your case you need to check what the command

UNIX95=YES ps -u ${ORACLE_ADMIN} -o args

returns and if the processes you are monitoring are in this list without any leading or trailing characters.

Your change would also work, but you possibly would run into trouble if the Oracle admin user issues a command like

$ vi /tmp/ora_reco_STUDUM.logfile

Your grep string without ^ and $ would identify above command as a valid process because it only greps for "ora_reco_STUDUM" and neglects that there are leading ("vi /tmp/") and trailing (".logfile") characters.

Carsten