topic Re: MySQL package does not fail over to standby node when primary node network link lost in Operating System - Linux

MySQL package does not fail over to standby node when primary node network link lost

tommy_28 — Tue, 22 Aug 2006 01:51:46 GMT

Hi,

Is there anyone can look up what happens for below problem?

Version: serviceguard-A.11.16.04-0
Platform: All
OS/Version: Red Hat

Description of problem:
MySQL package does not fail over to standby node when primary node network
link lost

Version or Release number of selected component (if n/a, use `uname -a`):
RHEL4 U2 , SG/LX 11.6x , MySQL 5.0.17c pro cert

How reproducible:

Steps to Reproduce:
1.Setup SG/LX
2.Install MySQL toolkit ,MySQL DB
3.Pull out LAN cable from primary node

Actual results:

MySQL package does not fail over to standby node. The cluster just got reformed and finally all nodes up, and the package does not failover.

#cmviewcl
Warning: Unable to send a datagram to the configuration daemon (cmclconfd) at 10.101.0.109: Input/output error

CLUSTER STATUS
cluster1 up

NODE STATUS STATE
rx1600-2 up running
rx1600-4 up running

PACKAGE STATUS STATE AUTO_RUN NODE
MySQL_PKG up running enabled rx1600-4

At the same time, /var/log/messages says:
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Obtaining Cluster Lock, sequence = 10
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to get lock /sg/cluster1. Lock
applicants: 2
Aug 22 14:41:22 rx1600-4 cmlocklund[6305]: Failed to obtain the lock!
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempt to get lock /sg/cluster1 unsuccesss
ful. Reason: lock_denied. Lock applicants: 2
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Cluster lock was denied. Lock was owned byy
another node.
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Attempting to adjust cluster membership
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Beginning standard partial election
Aug 22 14:41:28 rx1600-4 cmcld[6277]: Resumed updating safety time
Aug 22 14:41:28 rx1600-4 cmcld[6277]: 2 nodes have formed a new cluster, sequencc
e #3
Aug 22 14:41:28 rx1600-4 cmcld[6277]: The new active cluster membership is: rx166
00-2(id=1), rx1600-4(id=2)
Aug 22 14:46:13 rx1600-4 cmclconfd[12223]: cl_msg_udp_send: Input/output error

Expected results:
MySQL package fails over to standby node automatically and the database continues to function

Configuration:
lpcluster.config
NODE_NAME rx1600-2
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.109
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.20
CLUSTER_LOCK_LUN /dev/sdc1

NODE_NAME rx1600-4
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.146
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.40
CLUSTER_LOCK_LUN /dev/sdc1

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

Re: MySQL package does not fail over to standby node when primary node network link lost

melvyn burnard — Tue, 22 Aug 2006 02:24:40 GMT

so what subnet do you have in the package config file:
Also, are you using the internal built-in lans of these rx1600 servers for the SG config? If so, are they 100BT or 1000BT?

Re: MySQL package does not fail over to standby node when primary node network link lost

tommy_28 — Tue, 22 Aug 2006 02:31:10 GMT

The subnet as I wrote in the last part of previous msg.

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

The network cable I pulled is just the 10 subnet cable.

I used both of 1000BT and 100BT build-in NIC of rx1600, the first one is for 10 subnet, the other for 192 subnet. They're all configued as HEARTBEAT network, yet MySQL package use 10 subnet for data transfer.

Re: MySQL package does not fail over to standby node when primary node network link lost

melvyn burnard — Tue, 22 Aug 2006 04:43:34 GMT

You state:
mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

But what is the SUBNET configured in the package configuration file???
And you cannot use the built in 100BT LAN ports for Serviceguard on this model of server.

Re: MySQL package does not fail over to standby node when primary node network link lost

Steven E. Protter — Tue, 22 Aug 2006 06:29:28 GMT

Shalom,

Perhaps enable logging or set -x in the package control script. I'm wondering if its running or not when failover conditions are met.

SEP

Re: MySQL package does not fail over to standby node when primary node network link lost

tommy_28 — Thu, 07 Sep 2006 21:40:05 GMT

After config the subnet in the pkg config file,
the problem resovled.

Thanks for your help.