Serviceguard
cancel
Showing results for 
Search instead for 
Did you mean: 

MySQL package does not fail over to standby node when primary node network link lost

tommy_28
Frequent Advisor

MySQL package does not fail over to standby node when primary node network link lost

Hi,

Is there anyone can look up what happens for below problem?

Version: serviceguard-A.11.16.04-0
Platform: All
OS/Version: Red Hat

Description of problem:
MySQL package does not fail over to standby node when primary node network
link lost

Version or Release number of selected component (if n/a, use `uname -a`):
RHEL4 U2 , SG/LX 11.6x , MySQL 5.0.17c pro cert


How reproducible:


Steps to Reproduce:
1.Setup SG/LX
2.Install MySQL toolkit ,MySQL DB
3.Pull out LAN cable from primary node

Actual results:

MySQL package does not fail over to standby node. The cluster just got reformed and finally all nodes up, and the package does not failover.

#cmviewcl
Warning: Unable to send a datagram to the configuration daemon (cmclconfd) at 10.101.0.109: Input/output error

CLUSTER STATUS
cluster1 up

NODE STATUS STATE
rx1600-2 up running
rx1600-4 up running

PACKAGE STATUS STATE AUTO_RUN NODE
MySQL_PKG up running enabled rx1600-4


At the same time, /var/log/messages says:
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:18 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Obtaining Cluster Lock, sequence = 10
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to get lock /sg/cluster1. Lock
applicants: 2
Aug 22 14:41:22 rx1600-4 cmlocklund[6305]: Failed to obtain the lock!
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempt to get lock /sg/cluster1 unsuccesss
ful. Reason: lock_denied. Lock applicants: 2
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Cluster lock was denied. Lock was owned byy
another node.
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Attempting to form a new cluster
Aug 22 14:41:22 rx1600-4 cmcld[6277]: Beginning standard election
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Attempting to adjust cluster membership
Aug 22 14:41:27 rx1600-4 cmcld[6277]: Beginning standard partial election
Aug 22 14:41:28 rx1600-4 cmcld[6277]: Resumed updating safety time
Aug 22 14:41:28 rx1600-4 cmcld[6277]: 2 nodes have formed a new cluster, sequencc
e #3
Aug 22 14:41:28 rx1600-4 cmcld[6277]: The new active cluster membership is: rx166
00-2(id=1), rx1600-4(id=2)
Aug 22 14:46:13 rx1600-4 cmclconfd[12223]: cl_msg_udp_send: Input/output error

Expected results:
MySQL package fails over to standby node automatically and the database continues to function

Configuration:
lpcluster.config
NODE_NAME rx1600-2
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.109
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.20
CLUSTER_LOCK_LUN /dev/sdc1


NODE_NAME rx1600-4
NETWORK_INTERFACE eth0
HEARTBEAT_IP 10.101.0.146
NETWORK_INTERFACE eth1
HEARTBEAT_IP 192.168.0.40
CLUSTER_LOCK_LUN /dev/sdc1

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"
5 REPLIES
melvyn burnard
Honored Contributor

Re: MySQL package does not fail over to standby node when primary node network link lost

so what subnet do you have in the package config file:
Also, are you using the internal built-in lans of these rx1600 servers for the SG config? If so, are they 100BT or 1000BT?

My house is the bank's, my money the wife's, But my opinions belong to me, not HP!
tommy_28
Frequent Advisor

Re: MySQL package does not fail over to standby node when primary node network link lost

The subnet as I wrote in the last part of previous msg.

mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"

The network cable I pulled is just the 10 subnet cable.

I used both of 1000BT and 100BT build-in NIC of rx1600, the first one is for 10 subnet, the other for 192 subnet. They're all configued as HEARTBEAT network, yet MySQL package use 10 subnet for data transfer.
melvyn burnard
Honored Contributor

Re: MySQL package does not fail over to standby node when primary node network link lost

You state:
mysql.cntl:
IP[0]="10.101.1.152"
SUBNET[0]="10.0.0.0"


But what is the SUBNET configured in the package configuration file???
And you cannot use the built in 100BT LAN ports for Serviceguard on this model of server.
My house is the bank's, my money the wife's, But my opinions belong to me, not HP!
Steven E. Protter
Exalted Contributor

Re: MySQL package does not fail over to standby node when primary node network link lost

Shalom,

Perhaps enable logging or set -x in the package control script. I'm wondering if its running or not when failover conditions are met.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
tommy_28
Frequent Advisor

Re: MySQL package does not fail over to standby node when primary node network link lost

After config the subnet in the pkg config file,
the problem resovled.

Thanks for your help.