MySQL package does not fail over to standby node when primary node network link lost

[tommy_28]

Hi,

Is there anyone can look up what happens for below problem?

Version: serviceguard-A.11.16.04-0
Platform: All
OS/Version: Red Hat

Description of problem:
MySQL package does not fail over to standby node when primary node network
link lost

Version or Release number of selected component (if n/a, use `uname -a`):
RHEL4 U2 , SG/LX 11.6x , MySQL 5.0.17c pro cert


How reproducible:


Steps to Reproduce:
1.Setup SG/LX
2.Install MySQL toolkit ,MySQL DB
3.Pull out LAN cable from primary node

Actual results:

MySQL package does not fail over to standby node. The cluster just got reformed and finally all nodes up, and the package does not failover.

#cmviewcl
Warning: Unable to send a datagram to the configuration daemon (cmclconfd) at 10.101.0.109: Input/output error

CLUSTER STATUS
cluster1 up

NODE STATUS STATE
rx1600-2 up running
rx1600-4 up running

PACKAGE STATUS STATE AUTO_RUN NODE
MySQL_PKG up running enabled rx1600-4


At the same time, /var/log/messages says:
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Obtaining Cluster Lock, sequence = 10
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to get lock /sg/cluster1. Lock
applicants: 2
Aug 22 14:41:22 rx1600-4 cmlocklund[6305]: Failed to obtain the lock!
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempt to get lock /sg/cluster1 unsuccesss
ful. Reason: lock_denied. Lock applicants: 2
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Cluster lock was denied. Lock was owned byy
another node.
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Attempting to adjust cluster membership
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Beginning standard partial election
Aug 22 14:41:28 rx1600-4 cmcld[6277]: Resumed updating safety time
Aug 22 14:41:28 rx1600-4 cmcld[6277]: 2 nodes have formed a new cluster, sequencc
e #3
Aug 22 14:41:28 rx1600-4 cmcld[6277]: The new active cluster membership is: rx166
00-2(id=1), rx1600-4(id=2)
Aug 22 14:46:13 rx1600-4 cmclconfd[12223]: cl_msg_udp_send: Input/output error

Expected results:
MySQL package fails over to standby node automatically and the database continues to function

Configuration:
lpcluster.config
NODE_NAME rx1600-2
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.109
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.20
CLUSTER_LOCK_LUN /dev/sdc1


NODE_NAME rx1600-4
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.146
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.40
CLUSTER_LOCK_LUN /dev/sdc1

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

[melvyn burnard]

so what subnet do you have in the package config file:
Also, are you using the internal built-in lans of these rx1600 servers for the SG config? If so, are they 100BT or 1000BT?

[tommy_28]

The subnet as I wrote in the last part of previous msg.

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

The network cable I pulled is just the 10 subnet cable.

I used both of 1000BT and 100BT build-in NIC of rx1600, the first one is for 10 subnet, the other for 192 subnet. They're all configued as HEARTBEAT network, yet MySQL package use 10 subnet for data transfer.

[melvyn burnard]

You state:
mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"


But what is the SUBNET configured in the package configuration file???
And you cannot use the built in 100BT LAN ports for Serviceguard on this model of server.

[Steven E. Protter]

Shalom,

Perhaps enable logging or set -x in the package control script. I'm wondering if its running or not when failover conditions are met.

SEP

[tommy_28]

After config the subnet in the pkg config file,
the problem resovled.

Thanks for your help.