Failover problem after SAP BW upgrade

[Bruno Bossier_1]

We have a SAP BW system recently upgraded from 2.0 to 3.5. The SAP kernel went from 4.6D to 6.40. Since then, the failover from the original system to the other system in the cluster fails. See below part of the log.

Any ideas ?

Feb 20 15:44:19 - Node "uxwon02": (start_db): TRACE POINT
Feb 20 15:44:19 - Node "uxwon02": (start_db): Starting Database ...
Feb 20 15:44:19 - Node "uxwon02": (start_db): Database startup log is written to /home/pb1adm/startdb.log
Feb 20 15:44:32 - Node "uxwon02": (start_db): Database started successfully
Feb 20 15:44:32 - Node "uxwon02": (start_addons_postdb): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (start_addons_preci): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (start_ci): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (start_ci): Start Central Instance ...
Feb 20 15:44:32 - Node "uxwon02": (crit_start_app 10.6.64.74 11 pb1adm HP-UX r3 sleep 0; DVEBMGS): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (clean_ipc DVEBMGS 11 pb1adm): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (clean_ipc): Removing shmem of DVEBMGS11 on uxwon02 using normal cleanup policy
Feb 20 15:44:32 - Node "uxwon02": (clean_ipc): P: OsKey: 58900111 0x0382be8f SCSA Shared Memory Key: 58900000 removed
Feb 20 15:44:32 - Node "uxwon02": (start_app sgwon03 11 pb1adm HP-UX r3 sleep 0; DVEBMGS): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (is_ip_local sgwon03): TRACE POINT
Feb 20 15:44:32 - Node "uxwon02": (is_ip_local): sgwon03 is local
Feb 20 15:44:32 - Node "uxwon02": (start_app sgwon03 11 pb1adm): Startup attempt on local host...
Feb 20 15:44:32 - Node "uxwon02": (start_app sgwon03 11 pb1adm): Running /usr/sap/PB1/SYS/exe/run/startsap r3 DVEBMGS11
Feb 20 15:44:33 - Node "uxwon02": (watchdog): Watchdog initiated (PID: 1444 Timeout: 60 secs)
Feb 20 15:45:03 - Node "uxwon02": (start_app sgwon03 11 pb1adm): Finishing startup attempt
Feb 20 15:45:03 - Node "uxwon02": (test_app sgwon03 11 pb1adm DVEBMGS 2): TRACE POINT
Feb 20 15:45:03 - Node "uxwon02": (test_app): Trying to connect via RFC to host sgwon03 instance DVEBMGS11 ...
Feb 20 15:45:03 - Node "uxwon02": (watchdog): Watchdog initiated (PID: 1683 Timeout: 60 secs)
Feb 20 15:45:04 - Node "uxwon02": (test_app): No connection to instance DVEBMGS11
Feb 20 15:45:04 - Node "uxwon02": (test_app): Delaying 5 secs to allow instance startup/recover
Feb 20 15:45:09 - Node "uxwon02": (test_app): Trying to connect via RFC to host sgwon03 instance DVEBMGS11 ...
Feb 20 15:45:09 - Node "uxwon02": (watchdog): Watchdog initiated (PID: 1757 Timeout: 60 secs)
Feb 20 15:45:09 - Node "uxwon02": (test_app): No connection to instance DVEBMGS11
Feb 20 15:45:09 - Node "uxwon02": (test_app): Instance DVEBMGS11 not responding
Feb 20 15:45:09 - Node "uxwon02": (crit_start_app): WARNING: Non-zero exit status
ERROR: Function customer_defined_run_cmds
ERROR: Failed to RUN customer commands
Feb 20 15:45:09 - Node "uxwon02": (sapdbci_main): Entering SGeSAP stopDBCI runtime steps ...

[Geoff Wild]

Does it start manually?

Try starting the package by just mounting file systems.

Then start Oracle, then SAP.

If that works, then check your customer defined run commands - maybe that needs to be changed. Check with your Basis team on correct command to start SAP.

Rgds...Geoff

[Bruno Bossier_1]

Yes, manually all works fine. It is even so that the SAP fully starts, but Serviceguard thinks it didn't because it could not make a connection, hence the message :

Trying to connect via RFC to host sgwon03 instance DVEBMGS11 ...

[Geoff Wild]

Okay - so what is in your "customer defined run commands" section of the package control script?

Compare that with the command you use to start manually - maybe its a timing issue?

Rgds...Geoff

[Bruno Bossier_1]

We use the Serviceguard extensions for SAP. FYI :

/etc/cmcluster/PB1/sapdbci.cntl startDBCI PB1

[Geoff Wild]

That's how I start SAP as well...

Is there anything in the SAP log? should be:

/home/pb1adm/startsap_uxwon02_00.log

Can't connect via RFC - at first I was thinking something missing from /etc/services - but it runs manually - so that can't be it...

Rgds...Geoff

[Stuart Abramson]

What is host "sgwon03"? He's trying to connect there and can't. Is it a "hostname"? Is it a Virtual IP?

What is DVEBMGS11? Whatever it is, he thinks it's running on sgwon03. Is it?

[Bruno Bossier_1]

DVEBMGS11 is a standard SAP instance name. Each letter defines a functionality within SAP.

sgwon03 is a hostname for a virtual IP address which is created by Serviceguard when the package starts up.

Any more ideas ?

Bruno