Changing the Heartbeat IP in service guard

[Achille Costabile]

Hi,

I am trying change the Hearbeat IP of a 2 node cluster and it fails. It is lan6 on both nodes.

Here are the steps we tried to use to change the IP addresses.
ere are the steps we used to try and change the IP address:
node 1 - cmhaltnode -f
node 2 - cmhaltnode -f
Change the lan6 IP, Subnet & Broadcast setting for adl0715
Change the lan6 IP, Subnet & Broadcast setting for adl0716
Change the gmh_c11.ascii
- Under adl0715, below lan6, change the HEARTBEAT_IP
- Under adl0716, below lan6, change the HEARTBEAT_IP
reboot both servers
on adl0715, cmapplyconf -C gmh_c11.asciicmhaltnode -f
on adl0716, cmhaltnode -f
Change the lan6 IP, Subnet & Broadcast setting in netconf for node 1
Change the lan6 IP, Subnet & Broadcast setting in netconf for node 2
Change the cluster.ascii
- Under node 1, below lan6, change the HEARTBEAT_IP
- Under node 2, below lan6, change the HEARTBEAT_IP
reboot both servers
on node 1, cmapplyconf -C cluster.ascii
Then we get the error messgae I've attached. Cluster does not want to configure, and I am unable to rlogin to node 2

Please help?
AC

[Rajeev Shukla]

Hi AC,

Before you can re-configure your cluster you should make sure that you can rlogin/remsh to all nodes from each side.
i.e you should be able to rlogin from node1 to node2 and back from node2 to node1 (use /etc/hosts.equiv file to make that happen), make sure you have /etc/cmcluster/cmclnodelist file having the entries for both the nodes.
By doing that only your cmcheckconf -C will work and once things are fine you can go a head to cmapplyconf -C

Cheers
Rajeev

[Warren_9]

hi,

did you change the /etc/hosts after change the IP in netconf?

GOOD LUCK!!

[Achille Costabile]

Thanks for the reply.
I understand that, but I'm trying to determine why I was unable to rlogin/remsh between the nodes after changing the heartbeat IP. The hostname and primary IP stayed the same, so it should not have affected r-commands.
I did notice that when it is working with the old IP address, the default route points to lan1 (primary interface). When I changed the heartbeat lan, the default route IP remained the same, but it was pointing to lan6 (which is what I changed)
I'm just throwing everything up to see if I can nut out this problem

[Rajeev Shukla]

Hi AC,
Can you post your netstat -in and netstat -rn outputs from both the nodes after you have configured lan6 in /etc/rc.config.d/netconf

Rajeev

[Matthew_50]

before cmapplyconf, should always check setting by cmcheckconf.

Can you provide your each lan adapter detail setting ?

[Achille Costabile]

Hi Rajeev,

I didn't get the netstat for both nodes after the IP address change, only on 1 node
It is attached.

[Rajeev Shukla]

Hi AC,
I am bit confused with the way the interface card are configured on both of your nodes. Especially looking at lan6 on both the nodes, node has default route as 192.57.X.X and other has 148.95.X.X and looking at that, they will never be able to communicate. See ensure few things i guess..
1. Configure the interfaces in the /etc/rc.config.d/netconf correctly on each nodes and once done use linkloop from each node to see they can talk nicely. Once done, you can edit your cluster config file to have them there.

[Achille Costabile]

The 192 default route is before the lan6 change, and the 148 is after the lan6 change. These are on the same node, but on either side of the change window. I didn't get output from the other node

[Stephen Doud]

When no cluster exists, Serviceguard locates and documents all potential networks over which it can communicate with other nodes listed in a cmquerycl command. It then selects one of these LANs for internode communication from which SG derives sufficient information to build a cluster binary file.

When a cluster already exists (as recorded in the cluster binary file /etc/cmcluster/cmclconfig), SG expects to used the heartbeat NIC for inter-node communication. When the binary file is out of date with the new IP assignments, inter-node communication is not possible, generating the errors that you are seeing (unless another configuration change has occured).

Since you have already changed the IP addresses of your nodes, the easiest fix is to rm /etc/cmcluster/cmclconfig on each node and then "decluster" all volume groups (vgchange -c n ).
Next, do a cmquerycl to verify the current network configuration will support internode communication. If the resulting output imitates your modified cluster configuration ASCII file, then cmapplyconf your file. Else, edit the one that the new cmquerycl built - and cmapplyconf that one. Doing so will cmcheckconf the file and system devices, "cluster" all VOLUME_GROUPs listed in the file, install the cluster lock structure (if so designated) and distribute the new binary to all nodes in the cluster.