SAP Central Instance is not starting in DB node

[Bhagirath K Lader]

Hello Everyone:

I have SAP installed as two nodes cluster with nodes prspsci and prspsdb with two packages pkgciprs and pkgdbprs. While testing the failover, I found that pkgciprs package is not failing in node prspsdb.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The startsap_DVEBMGS00.log shows:


Trace of system startup/check of SAP System PRS on Sun Sep 6 04:25:29 IST 2009


Called command: /usr/sap/PRS/SYS/exe/run/startsap DVEBMGS00

Starting SAP Instance DVEBMGS00
------------------------------
Instance Service on host prspsdb started

SAP-R/3-Startup Program Rel 700 V1.8 (2003/04/24)
-------------------------------------------------

Starting at 2009/09/06 04:25:29
Startup Profile: "/usr/sap/PRS/SYS/profile/START_DVEBMGS00_prspsdb"

Update local Kernel Files
-------------------------
(16023) Local: /usr/sap/PRS/SYS/exe/ctrun/sapcpe name=PRS

Execute Pre-Startup Commands
----------------------------
(16023) Local: /usr/sap/PRS/SYS/exe/run/sapmscsa -n pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsdb
/usr/sap/PRS/SYS/exe/run/sapmscsa: make new mode. SCSA currently non existent.
sapcscsa: SCSA defined. sapscsaId == 779 == 0000030b
sapcscsa: SCSA attached at address c000000000067000
sapcscsa: SCSA initialized.
rslgwr1(21): Searching for overlap point in pre-existing SysLog file...
/usr/sap/PRS/SYS/exe/run/sapmscsa: finished.
(16023) Local: rm -f ms.sapPRS_DVEBMGS00
(16023) Local: ln -s -f /usr/sap/PRS/SYS/exe/run/msg_server ms.sapPRS_DVEBMGS00
(16023) Local: rm -f dw.sapPRS_DVEBMGS00
(16023) Local: ln -s -f /usr/sap/PRS/SYS/exe/run/disp+work dw.sapPRS_DVEBMGS00
(16023) Local: rm -f co.sapPRS_DVEBMGS00
(16023) Local: ln -s -f /usr/sap/PRS/SYS/exe/run/rslgcoll co.sapPRS_DVEBMGS00
(16023) Local: rm -f se.sapPRS_DVEBMGS00
(16023) Local: ln -s -f /usr/sap/PRS/SYS/exe/run/rslgsend se.sapPRS_DVEBMGS00

Starting Programs
-----------------
(16060) Starting: local ms.sapPRS_DVEBMGS00 pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsdb
(16061) Starting: local dw.sapPRS_DVEBMGS00 pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsdb
(16062) Starting: local co.sapPRS_DVEBMGS00 -F pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsdb
(16063) Starting: local se.sapPRS_DVEBMGS00 -F pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsdb
(16064) Starting: local /usr/sap/PRS/SYS/exe/run/igswd_mt -mode=profile pf=/usr/sap/PRS/SYS/profile/PRS_DVEBMGS00_prspsci
(16023) Waiting for Child Processes to terminate.
(16023) **** 2009/09/06 04:25:30 Child 16062 terminated with Status 1 . ****
(16023) **** 2009/09/06 04:25:46 Child 16061 terminated with Status 0 . ****
Instance on host prspsdb started

(16023) **** 2009/09/06 04:26:58 Caught Signal to Stop all Programs. ****

(16023) Parent Shutdown at 2009/09/06 04:27:00
(16023) Stop Child Process: 16064
(16023) Wait 300 sec for Process 16064 terminating...
(16023) Stop Child Process: 16063
(16023) Wait 300 sec for Process 16063 terminating...
(16023) Stop Child Process: 16060
(16023) Wait 300 sec for Process 16060 terminating...

Execute Post-Shutdown Commands
------------------------------
(16023) Exiting with Return-Code 0. (Ok)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The relevant portions of ciprs.cntl.log are:

########### Node "prspsdb": Starting package at Sun Sep 6 03:51:09 IST 2009 ###########
Sep 6 03:51:09 - Node "prspsdb": Activating volume group vgciSRM with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgciSRM" has been successfully changed.
Sep 6 03:51:09 - Node "prspsdb": Checking filesystems:
/dev/vgciSRM/lvsapPRS
fsck: /etc/default/fs is used for determining the file system type
file system is clean - log replay is not required
Sep 6 03:51:10 - Node "prspsdb": Mounting /dev/vgciSRM/lvsapPRS at /usr/sap/PRS/DVEBMGS00
Sep 6 03:51:10 - Node "prspsdb": Adding IP address 172.20.4.77 to subnet 172.20.4.0
Sep 6 03:51:10 - Node "prspsdb": (sapwas_main): Entering SGeSAP start runtime steps ...
Sep 6 03:51:10 - Node "prspsdb": (sapwas_main): /etc/cmcluster/PRS/sapwas.cntl: SGeSAP - B.04.00.00 - Last modified 08/10/05
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /etc/cmcluster.conf
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /etc/cmcluster/PRS/sap.config
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /opt/cmcluster/sap/sap.functions
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /etc/cmcluster/sap.functions
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /opt/cmcluster/sap/SID/customer.functions
Sep 6 03:51:10 - Node "prspsdb": (get_source): Found /etc/cmcluster/PRS/customer.functions
Sep 6 03:51:10 - Node "prspsdb": (initialize): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (check_version): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (check_parameters): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper 172.20.4.78): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper 172.20.4.78): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper 172.20.4.77): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (check_parameters): Package will handle SAP (ABAP) system central services
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (check_debug): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (check_access 172.20.4.78 /usr/sap/PRS/SYS/exe/ctrun): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (is_node_alive 172.20.4.78): TRACE POINT
Sep 6 03:51:10 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 4318 Timeout: 100 secs)
Sep 6 03:51:22 - Node "prspsdb": (check_access): NFS server host 172.20.4.78 for /usr/sap/PRS/SYS/exe/ctrun not responding - delayed
Sep 6 03:51:22 - Node "prspsdb": (check_access): Waiting 10 secs for NFS package to start
Sep 6 03:51:32 - Node "prspsdb": (is_node_alive 172.20.4.78): TRACE POINT
Sep 6 03:51:32 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 4988 Timeout: 100 secs)
Sep 6 03:51:34 - Node "prspsdb": (check_access): Testing NFS setup ...
Sep 6 03:51:34 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 5012 Timeout: 100 secs)
Sep 6 03:51:34 - Node "prspsdb": (check_access): /usr/sap/PRS/SYS/exe/ctrun accessible
Sep 6 03:51:34 - Node "prspsdb": (check_access 172.20.4.78 /usr/sap/trans/bin): TRACE POINT
Sep 6 03:51:34 - Node "prspsdb": (is_node_alive 172.20.4.78): TRACE POINT
Sep 6 03:51:34 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 5020 Timeout: 100 secs)
Sep 6 03:51:35 - Node "prspsdb": (check_access): Testing NFS setup ...
Sep 6 03:51:35 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 5196 Timeout: 100 secs)
Sep 6 03:51:35 - Node "prspsdb": (check_access): /usr/sap/trans/bin accessible
Sep 6 03:51:35 - Node "prspsdb": (app_handler start): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (check_own_app): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (stop_own_app 4): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (stop_other_inst): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (ci_remove_shmem normal DVEBMGS 00): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (clean_ipc DVEBMGS 00 prsadm): TRACE POINT
Sep 6 03:51:35 - Node "prspsdb": (clean_ipc): Searching shmem of DVEBMGS00 on prspsdb using normal cleanup policy
Sep 6 03:51:36 - Node "prspsdb": (clean_ipc): P: OsKey: 58900100 0x0382be84 SCSA Shared Memory Key: 58900000 removed
Sep 6 03:51:36 - Node "prspsdb": (app_remove_shmem): TRACE POINT
Sep 6 03:51:36 - Node "prspsdb": (check_db): TRACE POINT
Sep 6 03:51:36 - Node "prspsdb": (check_db): Checking executable directory
Sep 6 03:51:36 - Node "prspsdb": (check_db): Trying to connect to database using /usr/sap/PRS/SYS/exe/run/R3trans
Sep 6 03:51:39 - Node "prspsdb": (check_db): Waiting 10 secs for database to start
Sep 6 03:51:49 - Node "prspsdb": (check_db): Waiting 20 secs for database to start
Sep 6 03:52:10 - Node "prspsdb": (check_db): Database up and running
Sep 6 03:52:10 - Node "prspsdb": (start_addons_preci): TRACE POINT
Sep 6 03:52:10 - Node "prspsdb": (start_cs DVEBMGS): TRACE POINT
Sep 6 03:52:10 - Node "prspsdb": (start_cs): Start Central Instance ...
Sep 6 03:52:10 - Node "prspsdb": (crit_start_app 172.20.4.77 00 prsadm HP-UX r3 sleep 0; DVEBMGS): TRACE POINT
Sep 6 03:52:10 - Node "prspsdb": (clean_ipc DVEBMGS 00 prsadm): TRACE POINT
Sep 6 03:52:10 - Node "prspsdb": (clean_ipc): Searching shmem of DVEBMGS00 on prspsdb using normal cleanup policy
Sep 6 03:52:11 - Node "prspsdb": (start_app pkgciprs 00 prsadm HP-UX r3 sleep 0; DVEBMGS): TRACE POINT
Sep 6 03:52:11 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:52:11 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Startup attempt on local host...
Sep 6 03:52:11 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Moving old instance startup log to tmp
Sep 6 03:52:12 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Running /usr/sap/PRS/SYS/exe/run/startsap r3 DVEBMGS00
Sep 6 03:52:12 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 6625 Timeout: 100 secs)
Sep 6 03:52:43 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Information from SAP logfile /home/prsadm/startsap_DVEBMGS00.log:
Sep 6 03:52:43 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Instance on host prspsdb started
Sep 6 03:52:43 - Node "prspsdb": (start_app pkgciprs 00 prsadm): Finishing startup attempt
Sep 6 03:52:43 - Node "prspsdb": (test_app pkgciprs 00 prsadm DVEBMGS 5): TRACE POINT
Sep 6 03:52:43 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:52:43 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:52:43 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:52:43 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:52:43 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 7172 Timeout: 100 secs)
Sep 6 03:52:44 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:52:44 - Node "prspsdb": (test_app): Delaying 5 secs to allow instance DVEBMGS00 startup/recover
Sep 6 03:52:49 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:52:49 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:52:49 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:52:49 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:52:49 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 7380 Timeout: 100 secs)
Sep 6 03:52:50 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:52:50 - Node "prspsdb": (test_app): Delaying 10 secs to allow instance DVEBMGS00 startup/recover
Sep 6 03:53:00 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:53:00 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:53:00 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:53:00 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:53:00 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 7593 Timeout: 100 secs)
Sep 6 03:53:01 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:53:01 - Node "prspsdb": (test_app): Delaying 15 secs to allow instance DVEBMGS00 startup/recover
Sep 6 03:53:16 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:53:16 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:53:16 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:53:16 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:53:16 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 7834 Timeout: 100 secs)
Sep 6 03:53:17 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:53:17 - Node "prspsdb": (test_app): Delaying 20 secs to allow instance DVEBMGS00 startup/recover
Sep 6 03:53:37 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:53:37 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:53:38 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:53:38 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:53:38 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8069 Timeout: 100 secs)
Sep 6 03:53:38 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:53:38 - Node "prspsdb": (test_app): Instance DVEBMGS00 not responding
Sep 6 03:53:38 - Node "prspsdb": (crit_start_app): WARNING: Non-zero exit status
ERROR: Function customer_defined_run_cmds
ERROR: Failed to RUN customer commands
Sep 6 03:53:38 - Node "prspsdb": Halting service SAPMON
cmhaltserv : Service name SAPMON is not running.
Sep 6 03:53:38 - Node "prspsdb": (sapwas_main): Entering SGeSAP stop runtime steps ...
Sep 6 03:53:38 - Node "prspsdb": (sapwas_main): /etc/cmcluster/PRS/sapwas.cntl: SGeSAP - B.04.00.00 - Last modified 08/10/05
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /etc/cmcluster.conf
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /etc/cmcluster/PRS/sap.config
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /opt/cmcluster/sap/sap.functions
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /etc/cmcluster/sap.functions
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /opt/cmcluster/sap/SID/customer.functions
Sep 6 03:53:38 - Node "prspsdb": (get_source): Found /etc/cmcluster/PRS/customer.functions
Sep 6 03:53:38 - Node "prspsdb": (initialize): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (check_version): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (check_parameters): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper 172.20.4.78): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper 172.20.4.78): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper 172.20.4.77): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (check_parameters): Package will handle SAP (ABAP) system central services
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (ip_mapper): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (check_access 172.20.4.78 /usr/sap/PRS/SYS/exe/ctrun): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (is_node_alive 172.20.4.78): TRACE POINT
Sep 6 03:53:38 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8224 Timeout: 100 secs)
Sep 6 03:53:39 - Node "prspsdb": (check_access): Testing NFS setup ...
Sep 6 03:53:39 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8235 Timeout: 100 secs)
Sep 6 03:53:39 - Node "prspsdb": (check_access): /usr/sap/PRS/SYS/exe/ctrun accessible
Sep 6 03:53:39 - Node "prspsdb": (check_access 172.20.4.78 /usr/sap/trans/bin): TRACE POINT
Sep 6 03:53:39 - Node "prspsdb": (is_node_alive 172.20.4.78): TRACE POINT
Sep 6 03:53:39 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8243 Timeout: 100 secs)
Sep 6 03:53:40 - Node "prspsdb": (check_access): Testing NFS setup ...
Sep 6 03:53:40 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8254 Timeout: 100 secs)
Sep 6 03:53:40 - Node "prspsdb": (check_access): /usr/sap/trans/bin accessible
Sep 6 03:53:40 - Node "prspsdb": (stop_addons_preciapp): TRACE POINT
Sep 6 03:53:40 - Node "prspsdb": (app_handler stop): TRACE POINT
Sep 6 03:53:40 - Node "prspsdb": (check_own_app): TRACE POINT
Sep 6 03:53:40 - Node "prspsdb": (stop_own_app 2): TRACE POINT
Sep 6 03:53:40 - Node "prspsdb": (ci_remove_shmem normal DVEBMGS 00): TRACE POINT
Sep 6 03:53:40 - Node "prspsdb": (clean_ipc DVEBMGS 00 prsadm): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (clean_ipc): WARNING: shmem has processes attached
Sep 6 03:53:41 - Node "prspsdb": (app_remove_shmem): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (check_own_app): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_own_app 2): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_saposcol_app): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_addons_preci): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_cs DVEBMGS): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_cs): Halt Central Instance ...
Sep 6 03:53:41 - Node "prspsdb": (stop_app 172.20.4.77 00 prsadm HP-UX r3 DVEBMGS): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (is_ip_local 172.20.4.77): TRACE POINT
Sep 6 03:53:41 - Node "prspsdb": (stop_app 172.20.4.77 00 prsadm): Attempting local stop of instance...
Sep 6 03:53:41 - Node "prspsdb": (stop_app 172.20.4.77 00 prsadm): Running /usr/sap/PRS/SYS/exe/run/stopsap r3 DVEBMGS00
Sep 6 03:53:41 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8489 Timeout: 100 secs)
Sep 6 03:54:05 - Node "prspsdb": (ci_remove_shmem forced_strict DVEBMGS 00): TRACE POINT
Sep 6 03:54:05 - Node "prspsdb": (ci_remove_shmem): Removing shmem of 00 on prspsdb using forced_strict cleanup policy
Sep 6 03:54:05 - Node "prspsdb": (app_remove_procs DVEBMGS 00): TRACE POINT
Sep 6 03:54:05 - Node "prspsdb": (test_app 172.20.4.77 00 prsadm DVEBMGS 1): TRACE POINT
Sep 6 03:54:05 - Node "prspsdb": (is_ip_local 172.20.4.77): TRACE POINT
Sep 6 03:54:05 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:54:06 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed
Sep 6 03:54:06 - Node "prspsdb": (test_app): Trying to connect via RFC to host 172.20.4.77 instance DVEBMGS00 ...
Sep 6 03:54:06 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8981 Timeout: 100 secs)
Sep 6 03:54:06 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:54:06 - Node "prspsdb": (test_app): Instance DVEBMGS00 not responding
Sep 6 03:54:06 - Node "prspsdb": (wait_igs prsadm): TRACE POINT
Sep 6 03:54:06 - Node "prspsdb": (stop_cs): Collecting landscape info...
Sep 6 03:54:06 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 9092 Timeout: 100 secs)
Sep 6 03:54:06 - Node "prspsdb": (stop_addons_postci): TRACE POINT
Sep 6 03:54:06 - Node "prspsdb": (stop_saposcol): TRACE POINT
Sep 6 03:54:06 - Node "prspsdb": (stop_saposcol): Stopping /usr/sap/PRS/SYS/exe/run/saposcol on prspsdb
Sep 6 03:54:08 - Node "prspsdb": (sapwas_main): Leaving SGeSAP stop runtime steps
Sep 6 03:54:08 - Node "prspsdb": Remove IP address 172.20.4.77 from subnet 172.20.4.0
Sep 6 03:54:08 - Node "prspsdb": Unmounting filesystem on /dev/vgciSRM/lvsapPRS
umount: cannot unmount /usr/sap/PRS/DVEBMGS00 : Device busy
umount: return error 1.
WARNING: Running fuser to remove anyone using the file system directly.
/dev/vgciSRM/lvsapPRS: 6811co(prsadm)

Sep 6 03:54:09 - Node "prspsdb": Deactivating volume group vgciSRM
Deactivated volume group in Exclusive Mode.
Volume group "vgciSRM" has been successfully changed.
###### Node "prspsdb": Package start failed at Sun Sep 6 03:54:09 IST 2009 ######

[melvyn burnard]

Well the log file shows:

Sep 6 03:53:17 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00
Sep 6 03:53:17 - Node "prspsdb": (test_app): Delaying 20 secs to allow instance DVEBMGS00 startup/recover
Sep 6 03:53:37 - Node "prspsdb": (is_ip_local pkgciprs): TRACE POINT
Sep 6 03:53:37 - Node "prspsdb": (test_app): Trying to connect local dispatcher of instance DVEBMGS00 ...
Sep 6 03:53:38 - Node "prspsdb": (test_app): dpmon:
connection to dispatcher failed <<<<<<<<
Sep 6 03:53:38 - Node "prspsdb": (test_app): Trying to connect via RFC to host pkgciprs instance DVEBMGS00 ...
Sep 6 03:53:38 - Node "prspsdb": (watchdog): Watchdog initiated (PID: 8069 Timeout: 100 secs)
Sep 6 03:53:38 - Node "prspsdb": (test_app): No connection to instance DVEBMGS00 <<<<<<<<<<<
Sep 6 03:53:38 - Node "prspsdb": (test_app): Instance DVEBMGS00 not responding <<<<<<<<<<
Sep 6 03:53:38 - Node "prspsdb": (crit_start_app): WARNING: Non-zero exit status



So it appears hat yo do not have a dispatch process connection, or something along those lines.
I would raise a call with SAP

[Bhagirath K Lader]

Thanks for the response.


A Call to HP has been logged. However, we have not got any solution from HP as of now.

Regards
Bhagirath

[Hein van den Heuvel]

>> A Call to HP has been logged.

Now WHY would you do that?

The initial attack point is that this is an APPLICATION setup or configuration problem which is best handle by the application support folks. That would be your friendly SAP support organization best I can tell. If they haver reason to believe something in HPUX is broken then they'll let you know, with the appropriate specifics.

While you get support, you should review all you system and servives names, and IP configurations and port setttings the etc/services services files, and further make sure they match the SAP configurations in the PROFILES.
- Support will want to know
- That is probably where the problem is.

Finally, be sure tot get the dev_ log files for obvious errors. Again, support will want to see those anyway.

hth,
Hein.