Serviceguard Test Suite Whitepaper 

DOCUMENT
To test Serviceguard, it is necessary to understand what trigger events
Serviceguard sees as failures and how Serviceguard will react to these
trigger events.  Serviceguard accommodates the following failure modes with
the corresponding remedy.

Failure detected (Trigger event):  Node outage:  power failure,
complete heartbeat failure, reboot, system hang

Recovery Action:  Serviceguard causes the remaining node(s) to reform
the cluster, excluding the unresponsive node from the cluster, and possibly
taking over operation of its' dead packages.  A Serviceguard feature causes a
hanging node to reboot itself to prevent unintended I/O writes in the event
of it's re-awakening after another node has adopted it's packages and data.

Failure detected: Heartbeat transmission failure between nodes
Details:
If the cluster has more than one node, at least one LAN is used for
Serviceguard inter-node communication and heartbeat transmission.  This LAN
is called the heartbeat LAN.  If heartbeat transmission ceases between nodes
because of a node power failure, crash or hang, or a breakdown of all
heartbeat networks, cluster reformation by the surviving nodes is warranted.

Recovery action:
A nodes' reaction to the loss of heartbeat traffic from the other node(s) is
determined by the following cluster reformation rules.

Rule 1)
If the cluster is halved by a heartbeat network breakdown, a representative
of each half will seek authorization from the cluster arbitration device
(cluster lock disk or quorum server) to form a reduced cluster.  The half
which gets to the arbitration device last is forced by Serviceguard code to
reboot (actually transfer control (TOC) to the dump-saving firmware code -
see isl(1m)) to avoid the potential for data corruption.  This is necessary
because the node(s) which received authorization to reform a reduced cluster
by reaching the arbitration device first cannot determine the status of the
nodes on the other side of the HB network breakdown.  Therefore the surviving
nodes may immediately begin package take-over to reduce application
downtime.

Rule 2)
If the heartbeat transmission failure segregates a minority of nodes in the
cluster from remaining nodes (ex: 1 node from 3 nodes), the node finding
itself in the minority will TOC itself, while the nodes in a majority will
reform a reduced cluster and may adopt packages from the TOC'd node.

Failure detected: LAN adapter failure

Recovery action:  If configured, Serviceguard switches in a standby
NIC (Network Interface Card/Adapter) - usually within 2-3 seconds.  This
causes only a temporary network pause from the perspective of the users.

Unless the failed network is the sole heartbeat LAN (see above) Serviceguard
will do nothing and leave the LAN unresponsive or it will follow the SUBNET
failure protocol reviewed next.

Failure detected: SUBNET failure

Recovery action: The administrator may configure a SUBNET dependency
for a package.  In such a case, should the subnet (both primary and its
standby LAN) fail to transmit, Serviceguard will halt the package and
start(*) the package on an adoptive node that maintains subnet activity.

Failure detected: Package or package SERVICE failure
Details:
A package failure occurs when it's host server is no longer able to operate
the package, either due to the host node's failure or an optional package
dependency not being met.  Optional dependencies include SUBNET, HA-Monitor
RESOURCE, or SERVICE processes.  These optional dependency parameters are set
in the package configuration file.

An example of a SERVICE would be a monitor script which exits when an Oracle
system process is no longer detected.

Recovery Action:
A failure of an optional dependency will cause Serviceguard to halt the
package on the current node (or TOC the current node, depending on the
configuration of FAIL_FAST parameters) and start(*) the package on an
adoptive node.

Cluster autostart
Though not a failure mode, Serviceguard enables a node to join a running
cluster after reboot, or form a cluster with all cluster member nodes which
are likewise attempting to form a cluster after reboot.

Serviceguard Test Suite

This test suite demonstrates Serviceguard reactions to the above failure
modes (with the exception to optional HA-Monitor product package RESOURCE
triggers).

Serviceguard failure handling mechanisms are discussed conceptually to
provide predictability in the results of the testing.

NOTE:  certain tests configurations require the cluster binary file
acknowledge the test configuration.  Whether modifying a cluster
configuration file or package configuration file to meet these requirements,
a subsequent cmapplyconf is necessary to update the cluster binary file.  The
cluster (or package) may have to be down in order to perform the
reconfiguration.

### NODE OUTAGE
Cluster reformation detection:
Serviceguard utilizes a network-based heartbeat transmission to ascertain the
vitality of each node in the cluster.  When heartbeat transmission ceases
from any node in the cluster, a cluster reformation is triggered.  After
following cluster protocol (see below), each node that is able to respond
broadcasts a reformation message, takes votes from responding nodes and
determines a new cluster and package manager.

The reformation period is determined by several factors influenced
predominantly by the NODE_TIMEOUT parameter specified by the cluster
configuration file.  The reformation time includes a grace time interval to
allow the non-responding node(s) time to respond and request admittance to
the reforming cluster.

Serviceguard is designed to use the following protocol response after
broadcasting a cluster reformation message in order to determine the members
of the new cluster:

Cluster Reformation Protocol 1
- When half of the nodes fail to respond, seek the cluster arbitration device
  for permission to reform the cluster.  This is necessary in the event that
  all nodes are still alive, but the heartbeat transmission medium has
  failed.  Arbitration prevents a split-brain situation where subclusters
  form, adopting each others "orphaned" packages, and corrupting package
  data.  The first node that receives authorization from the arbitration
  device is permitted to reform the cluster.  The late node must TOC
  (Transfer Of Control) - meaning save a memory dump and reboot.

  This can be demonstrated in a 2-node cluster, where heartbeat (HB)
  transmission capability ceases.  Regardless of where the HB network failure
  occurs, one node will get permission to continue cluster operations, while
  the other one reboots itself.  Without the serial heartbeat function
  enabled, it is possible for a network-disabled node to form the new
  cluster, whilst the network-enabled node reboots itself.  When a serial
  (RS232) HB cable is configured, Serviceguard is allowed to continue
  operation, eventually determining the health of each NIC it oversees.  If
  all NICS are unable to transmit, that node TOCs.

  This protocol allows testing of the cluster arbitration device.  A cluster
  arbitration device is either a cluster lock disk, or a quorum server (on
  Serviceguard version 11.14 or later clusters).

Cluster Reformation Protocol 2
- When more than half of the nodes fails to respond, the minority set of
  nodes must TOC.
  Example:  when a HB transmission failure occurs between 1 node and 2 other
  nodes in a 3-node cluster, the single node determines itself to be in a
  minority - it TOCs itself, regardless of the state of the other 2 nodes.

Cluster Reformation Protocol 3
- When less than half of the nodes fails to respond, the majority nodes
  automatically reform the cluster.
  Example:  when a HB transmission failure occurs between 1 node and 2 other
  nodes in a 3-node cluster, the 2 nodes still communicating with one another
  determine they are in a majority and automatically reform the cluster.

Test Scenarios - Node outage
----------------------------
A node outage can be simulated by
- removing power from a server.
- disconnecting all heartbeat transmission cables on a server.
- performing a reboot command, which bypasses the cmhaltnode procedure
  included in the graceful shutdown scripts.

Expected test outcome:
Serviceguard should cause remaining active nodes in the cluster to reform the
cluster, and optionally take over orphaned packages from the dead node.
Before proceding with this test, consider incorporating package failover
testing.

For a 2 and 4-node cluster, half of the nodes will remain active, and form a
new cluster, while the other half will reboot.

For a 3-node cluster, a general HB network outage will induce all nodes to
reboot (protocol 2).  A HB outage between 1 and 2 nodes will result in the
single-node rebooting itself, and the remaining 2 nodes forming a cluster.

If serial-heartbeat is configured, loss of all HB network transmission on one
node should see that node TOC.

Note:  before proceding, verify (cmviewcl) that the cluster is running on
at least 2 nodes in the cluster.

### LAN ADAPTER FAILURE/RECOVERY
Serviceguard is capable of transferring the network stack for a failed LAN
NIC to a standby NIC residing on the same bridged network, limiting network
outage time to the NETWORK_POLLING_INTERVAL time specified in the cluster
configuration file - nominally 2 seconds.  When the primary NIC has
recovered, Serviceguard automatically transfers network traffic back to the
primary LAN.  "cmviewcl -v" output and syslog.log report LAN state.  If the
standby LAN fails, no transfer occurs.  Only cmviewcl and syslog.log register
the failure.

CAVEATS FOR A STANDBY LAN:
C1. A standby NIC MUST share the same bridged network with the primary.
  The cluster configuration file reports this relationship in the comments
  below each node's section.  Example: 

# Possible standby Network Interfaces for lan0: lan1.

   cmviewconf reports the same bridged network:

# cmviewconf | grep -i -e node -e lan -e bridge 
   node timeout:                        8.00    (seconds)
Cluster Node information:
   Node ID 1:
      Node name:                        eon
        network interface name:         lan0
        bridged net ID:                 1
        network interface name:         lan1
        bridged net ID:                 1   <-------
   Node ID 2:
      Node name:                        ion
        network interface name:         lan0
        bridged net ID:                 1
        network interface name:         lan1
        bridged net ID:                 1   <------- 

C2. The standby LAN cannot be listed in /etc/rc.config.d/netconf, and must have
  no IP assigned to it.

C3. Where network devices such as switches or hubs are used to link the primary
  and standby NIC, good high-availability practice requires that the standby
  NIC be attached to a different network device to avoid the potential for a
  single-point of failure.  Failure to do so voids the value of redundancy at
  the single point of failure.

C4. Local switching (between two network cards on the same machine) is only
  supported between network interface cards of the same type.  More
  specifically, local switch works between NICs that have the same network
  stack.  For example, all Ethernet cards use the same network stack, so they
  can be used in any combination for local switching.  This means that
  network interface cards such as 10Base-T, 100Base-T, 100VGAnyLan,
  1000Base-T and 1000Base-SX can be combined for local switching.  An example
  of combinations of cards that cannot be used for local switching are
  Ethernet cards with FDDI, Token ring, ATM or Fibre Channel cards, since all
  of these cards use different network stacks.

C5. Current versions of Serviceguard provide standby failover for the following
  network types:  Ethernet, FDDI, token ring.

C6. Serviceguard does not support the following for heartbeat transmission
  or standby failover:  ATM, Hyperfabric, PSI - Programmable Serial
  Interface, X.25.  Special monitors can be written to detect failure and
  invoke standby failover, but such is beyond the scope of this document.

Test Scenarios - LAN Adapter failure
------------------------------------
Prior to testing, verify both primary and standby LAN are "up":
Example: # cmviewcl -v 

    ....
    Network_Parameters:
    INTERFACE    STATUS       PATH         NAME
    PRIMARY      up           24.1         lan0
    STANDBY      up           56.1         lan1 

NOTES:
a.  A failure state registers as "down".
b.  Use "netstat -in" to determine which LAN is handling the IP.
c.  A cluster will not form if the HB network NICs are disabled. 

LAN failover can be produced by
- removing network cables
- powering down network components linking nodes together
- dead network adapter  (see note c. above)

A LAN failure CANNOT be simulated by using "ifconfig" to "down" a network
card.  ifconfig works at the TCP/IP layer 3 of the OSI model, while
Serviceguard does NIC validation at the link level (layer 2).  Hence, an
ifconfig does not disable link level traffic.

Expected test outcome:
- A failure state registers as "down" in cmviewcl -v output.
- "netstat -in" shows the IP being handled by the standby LAN
- /var/adm/syslog/syslog.log contains messages that indicate NIC failure and
  subsequent recovery.

LAN recovery
- replace the LAN cable
- power-up the network device linking nodes

Expected test outcome:
- "cmviewcl -v" status shows LAN "up"
- "netstat -in" shows traffic back on primary NIC
- syslog.log tracks/reports primary NIC back online

### SUBNET FAILURE
In order to test this scenario, a Serviceguard package configuration must be
configured with a legitimate subnet entry:

SUBNET                  16.113.0.0

The package configuration file must then be "cmapplyconf"'d into the cluster
binary file.

The cluster binary will prove it:

# cmviewconf | grep -e "package name" -e "package subnet"
      package name:                     clk
      package subnet:                   16.113.0.0

   "cmviewcl -v" will report a subnet under monitor:

   ...
    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    clk          up           running      disabled     eon

      Policy_Parameters:
    ...

      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME
      Service    up                  0         0   clock
      Subnet     up                                16.113.0.0 <<<--- HERE

Serviceguard tests all monitored LANs every NETWORK_POLLING_INTERVAL as
stated before.  If Serviceguard detects both PRIMARY and STANDBY NICs cannot
operate a subnet, it will halt a package dependent on that subnet.  (Note
that a standby LAN need not exist for subnet failure detection to
operate.)

If an adoptive node is enabled to take the package, and that node has access
to the subnet, Serviceguard will start the package on that node.

If no node has access to the subnet, Serviceguard will leave the package down
until a polling interval reveals the subnet is active on a node - and will
auto-start the package on that node.

Test Scenarios - SUBNET failure
-------------------------------
A node-specific subnet failure can be simulated by:
- disconnecting both primary and if configured, standby NIC cables
- disable network equipment providing access to the subnet

Expected outcome:
- Serviceguard will halt the package on the primary node
- Serviceguard may start the package on an adoptive node if global and
  node switching are "enabled"

- "cmviewcl -v" will report the subnet "down"
or
  the package starting/up on an adoptive node

- syslog.log will register the subnet failure and subsequent halting of the
  package.

A global subnet failure can be simulated by:
- disconnecting all network cables supporting the subnet
- disabling all network access to the subnet

Expected outcome:
- The package will halt.
- cmviewcl will show the subnet "down".
- syslog.log will report the subnet down.

A subnet recovery can be simulated by:
- reconnecting primary and standby NIC cables
- reenabling network access to the subnet

Expected outcome:
If a global subnet outage has occured, the package will autostart on the
first node regaining subnet transmission capabilities.

If the package FAILBACK_POLICY is set to MANUAL, the package will remain
running on the adoptive node.

If the package FAILBACK_POLICY is set to AUTOMATIC, the package will transfer
back the primary node (assuming the adoptive node was running it).

### PACKAGE SERVICE FAILURE
In order to testing of a SERVICE failure, a SERVICE must be configured
in a package configuration file and referenced in it's package control
script:

# grep ^SERVICE pkg.conf control.sh
pkg.conf:SERVICE_NAME                   clock
pkg.conf:SERVICE_FAIL_FAST_ENABLED      NO
pkg.conf:SERVICE_HALT_TIMEOUT           100
control.sh:SERVICE_NAME[0]="clock"
control.sh:SERVICE_CMD[0]="/usr/bin/X11/xclock -display 192.10.25.54:0"
control.sh:SERVICE_RESTART[0]=""

   The cluster binary must prove the configured SERVICE:
# cmviewconf | grep -i service
      package services:
        service ID:                     1
          service name:                 clock
          service halt timeout:         100     (seconds)
          service fail fast:            Disabled

SERVICE discussion:
A service is a dependency that the package requires in order to operate
properly.  For example, an Oracle ora_lgwr_ process is necessary for
Oracle to operate properly.  Therefore, the SERVICE_CMD could call a script
that checks for the existence of ora_lgwr_.  If the process dies, the
script that checks for the process should exit, signalling a failure of the
service to Serviceguard.  In the above example, if the xclock widget is
killed, the service is terminated and Serviceguard will respond.  The
SERVICE_RESTART parameter can be set to restart the service repeatedly
ad-infinitum, restart it a number of times, or not restart it at all.

Test Scenarios - SERVICE failure
--------------------------------
A SERVICE failure can caused by:
- killing the process indicated in the SERVICE_CMD statement in the package
  control script
- killing a process that is monitored by the script called in the SERVICE_CMD
  statement

Expected test outcome:
- package halt, when no adoptive node is available
- package failover, when an adoptive node is available and enabled to take
  the package

- When SERVICE_FAIL_FAST_ENABLED is DISABLED in the package configuration
  file (and cluster binary) and the service dies, the node will not TOC
  itself.  When ENABLED, the node will TOC itself if the service dies.

- syslog.log will record a SERVICE failure

   See the discussion on enabling a node to take over a package in the
   section titled "PACKAGE FAILOVER TESTING"

### Package and Node Fail-Fast Testing
See the discussion in the previous SERVICE testing section and a
standard package configuration file regarding SERVICE_FAIL_FAST_ENABLED

The NODE_FAIL_FAST_ENABLED parameter in a package can be tested to see if a
node will TOC when the package fails.  This is useful when a package fails to
start properly on a node and it is essential that the node reboot in order to
force a package halt/failover.

Though available, few clusters actually have a requirement to implement this
feature.

### PACKAGE FAILOVER TESTING
Consequent to Serviceguard detecting a failed node, subnet or package
service, package failover to an adoptive node is an essential task to insure
high availability for critical applications.  Hence, this section is devoted
to explaining how package switching is enabled should the need arise for a
package to move to an adoptive node.

Discussion:  enabling a node to take over a package:  In order for a node to
be a host for a package, it must meet these qualifications:

- have access to all the resources needed to operate the package on the
  primary node
- be configured as a standby LAN in the package configuration file (& cluster
  binary)
- be "enabled" to run the package
  There are 2 kinds of package switching enablement:
  o Global switching - (AUTORUN in cmviewcl -v -p [pkgNAME])
  o Node-switching

  Use cmviewcl to see both types.  Example: 

# cmviewcl -v -p clk | egrep -i 'eon|ion|node'
    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    clk          up           running      disabled     eon
      Failover        configured_node
      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      eon          (current)
      Alternate    up           enabled      ion 

To enable global switching for a package, use this command syntax:
cmmodpkg -e <package name> 

To enable node switching for a package, use this command syntax:
cmmodpkg -e -n <nodename> <package name> 

Test scenarios - Package failover testing
-----------------------------------------
A package failover can be caused by:
- cmhaltnode - causes packages running on the node to failover
- killing a node running a package
- disabling a subnet that a package is dependent on
- killing a service that the package is dependent on

Expected test outcome:
- cmviewcl reports that package failover occured if global and node-switching
  are enabled
- syslog.log records the package failover
- package control log records the package halt or start (depending on the
  node).

### CLUSTER AUTOSTART
A cluster auto-start after reboot is tangeantly related to failures in a
Serviceguard cluster.  The system administrator often requires a rebooting
node to rejoin a running cluster in order to be available to takeover or
start packages.

The autostart feature is not well understood.  A common misconception is that
a node will start a cluster by itself when partner nodes are not up, or not
running Serviceguard.  Such is not the case.  The following discussion
explains the mechanism that enables a node to join a running cluster or form
a cluster.

The AUTOSTART_CMCLD parameter in the /etc/rc.config.d/cmcluster file
must be set to "1" in order to enable this feature.

When set to 1, /sbin/init.d/cmcluster is launched via this run-time link:
/sbin/rc3.d/S800cmcluster

The crucial command launched in /sbin/init.d/cmcluster that enables
auto-start or join is the "cmrunnode" command.

cmrunnode has two actions:
1- join a currently running cluster.
   This action should take only seconds if a Serviceguard is already running
   on at least one other member node of the cluster.

2- form a new cluster
   This second action can only succeed when ALL other member nodes are also
   attempting to form a cluster!!  A caveat is that they have not hit the
   AUTOSTART_TIMEOUT limit discussed below.

A timeout time limit for the cmrunnode command is set in the cluster
configuration file (and cluster binary).  The parameter is called
AUTOSTART_TIMEOUT.  Typically, AUTOSTART_TIMEOUT is left at 10 minutes.  When
cmrunnode hits the expiration time, cmrunnode exits, and leaves a message in
syslog.log indicating that the timeout has occured.  If multiple nodes are
rebooted, each node must launch a cmrunnode with AUTOSTART_TIMEOUT minutes of
each other to successfully form a cluster.

It is also important to note that when a cmrunnode is waiting for a cluster
formation or join to occur, no other cmrunnode or cmruncl commands will
execute, as they will be queued up to run after the primary cmrunnode dies.

Test Scenarios - Cluster Autostart
----------------------------------
A test of the cluster-join action can be performed by following this
sequence:

- set  /etc/rc.config.d/cmcluster: AUTOSTART_CMCLD=1
- insure at least one other node is running Serviceguard (cmcld).
- perform a cmhaltnode to leave a currently running cluster
- perform a "shutdown -ry"

Although a node may rejoin a cluster after rebooting, it won't
necessarily automatically start a package unless
- that node is the primary node to operate the package
and
- the package is enabled to start on the node
and
  - the package is not running on any other node
  or
  - Package FAILBACK_POLICY is set to AUTOMATIC

Expected test outcome:
- The node will automatically join the running cluster after reboot
- syslog.log will record the process, including a cluster reformation

A test of the cluster-formation action can be performed by following this
sequence:

- set /etc/rc.config.d/cmcluster:  AUTOSTART_CMCLD=1 on all nodes
- perform a "cmhaltcl -f" to stop the cluster
- perform a "shutdown -ry" on all nodes
NOTE:  the time window for all nodes to execute the
     /sbin/init.d/cmcluster boot-time script must not exceed the
     AUTOSTART_TIMEOUT value specified in the cluster configuration
     file and binary (as compiled from the cluster configuration ASCII
     file).

Expected test outcome:
- All nodes form a cluster, and all packages configured to start at cluster
  formation time will start.
- syslog.log records cluster formation on each node
- package logs record package startup
- cmviewcl reports cluster is UP.

NOTES:
(*) provided AUTO_RUN and Node_switching are enabled.

                           ### DOCUMENT END ###