Cluster node reboots when it should failover

[Eric Herr]

I am testing a 2 node cluster with one package configured. I can manually move the package to both nodes without any problems. However, when I pull the primary lan interface on the primary node the package does not failover. Instead the failover node reboots probably to prevent a split-brain scenerio. Why does the package not failover when the primary interface is lost.

I'm attaching my ascii config file. Also, this is the last few lines of syslog.log on the system that reboots.

Mar 27 12:36:00 gobhup11 cmcld[15662]: Obtaining Cluster Lock
Mar 27 12:36:01 gobhup11 cmcld[15662]: Cluster lock was denied. Lock was obtained by another node.
Mar 27 12:36:01 gobhup11 cmcld[15662]: Attempting to form a new cluster
Mar 27 12:36:01 gobhup11 cmcld[15662]: Beginning standard election
Mar 27 12:36:11 gobhup11 cmcld[15662]: Cluster lock has been denied
Mar 27 12:36:52 gobhup11 cmcld[15662]: Service cmfileassistd terminated due to an exit(0).

[IT_2007]

please also post package config and control file. Seems something preventing in package config file.

[Sundar_7]

Eric,

When you pull the primary lan interface if the systems were able to communicate the heartbeat on lan2, then the expected behaviour is to halt the package and start it in the adoptive node.

Since this is not happening, my initial suspicion is on the second heartbeat network on lan2.

Did you let the cmquerycl create the cluster ascii file or did you manually edit the entries ?

You could try a linkloop between the two nodes and confirm the link level connectivity between the nodes.

Sundar.

[Eric Herr]

Attached is the package config

[Eric Herr]

I did use cmquerycl and am using ServiceGuard Manager to manage the cluster. I also do have 2 heartbeats configured (server primary and dedicate link for heartbeat) but my assumption is that losing either heartbeat would initiate a cluster failover. THanks.

[IT_2007]

If cluster and package running, then run cmquerycl and capture output and see what it shows.

If there is no problem in ascii file then it would be network issue. Did you tried to pull data lan cable and see how it behaves?

[Eric Herr]

I ran the cmcheckconf to verify my ascii config file. It states verification completed with no errors but also gives me several messages that state "The volume group ???? is active on more than one node". I have verified this and this is not the case. Any idea why I am seeing this message?

[melvyn burnard]

First thing I notice is that there is no SUBNET parameter set in the package config.
This means that the package will NOT be monitoring any SUBNET for failures that woul dcause it to switch in the event of a subnet failing.

Secondly, you need to check the OLDsyslog.log of the node that TOC'ed to see what it logged, as well as check further into the surviving node's syslog to see what it registerd for the netwroking.
Also, when the cluster is started it will log configuration information in the syslog.log, can you locate that and post it?
You definitely appeared to have lost heartbeats here, as it went for the cluster lock disc.
Can you run cmscancl and post the output?