Serviceguard package ip failover to standby NIC not happening only NODE IP fails over

[Dan Ryan_4]

Hello,

I have a problem with Serviceguard Primary to Standby failover.
The failover for the primary lan0 to standby lan2 happens for the node IP but not the package IP on lan0:1 .

So when we disable the LAN0 Network switch port and fail onto lan2 we can access the Node (10.20.209.25) but not the package (10.201.209.23).
Is there some SG configuration or OS configuration needed to have lan0:1 failover to lan2:1 ?

HPUX11i-OE-MC.ServiceGuard            A.11.16.00     ServiceGuard

wilerpa:/etc/cmcluster# netstat -in
Name      Mtu  Network         Address         Ipkts   Ierrs Opkts   Oerrs Coll
lan0:1    1500 10.201.209.0    10.201.209.23   283647247 0     7467815 0     0    ===> package
lan2*     1500 none            none            0       0     0       0     0
lan0      1500 10.201.209.0    10.201.209.25   5510277 0     319393526 0     0    ====> node
lo0       4136 127.0.0.0       127.0.0.1       5977333 0     5977344 0     0
lan4      1500 192.168.100.0   192.168.100.1   361352  0     184021  0     0

Here is the cmquerycl -w full output

Node Names:    wilerpa
               wilerpb

Bridged networks (full probing performed):

1       lan0           (wilerpa)
        lan2           (wilerpa)
        lan0           (wilerpb)
        lan2           (wilerpb)

2       lan4           (wilerpa)
        lan4           (wilerpb)

IP subnets:

IPv4:

10.201.209.0       lan0      (wilerpa)
                   lan0      (wilerpb)

192.168.100.0      lan4      (wilerpa)
                   lan4      (wilerpb)

IPv6:

Possible Heartbeat IPs:

10.201.209.0                      10.201.209.25       (wilerpa)
                                  10.201.209.26       (wilerpb)

192.168.100.0                     192.168.100.1       (wilerpa)
                                  192.168.100.2       (wilerpb)

Here is the cmviewcl -v output

CLUSTER      STATUS
erpprd       up

  NODE         STATUS       STATE
  wilerpa      up           running

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      up           0/1/2/0             lan0
    PRIMARY      up           0/6/1/0/6/0         lan4
    STANDBY      up           0/2/1/0/6/0         lan2

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    wilerpk1     up           running      enabled      wilerpa

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME
      Subnet     up                                10.201.209.0

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      wilerpa      (current)
      Alternate    up           enabled      wilerpb

  NODE         STATUS       STATE
  wilerpb      up           running

    Network_Parameters:
    INTERFACE    STATUS       PATH                NAME
    PRIMARY      up           0/1/2/0             lan0
    STANDBY      up           0/2/1/0/6/0         lan2
    PRIMARY      up           0/6/1/0/6/0         lan4

    PACKAGE      STATUS       STATE        AUTO_RUN     NODE
    wilerpk2     up           running      enabled      wilerpb

      Policy_Parameters:
      POLICY_NAME     CONFIGURED_VALUE
      Failover        configured_node
      Failback        manual

      Script_Parameters:
      ITEM       STATUS   MAX_RESTARTS  RESTARTS   NAME
      Subnet     up                                10.201.209.0

      Node_Switching_Parameters:
      NODE_TYPE    STATUS       SWITCHING    NAME
      Primary      up           enabled      wilerpb      (current)
      Alternate    up           enabled      wilerpa
wilerpa:/etc/cmcluster#

[Matti_Kurkela]

Please run:

cmgetconf -p wilerpk1 /tmp/wilerpk1.ascii
cmgetconf -p wilerpk2 /tmp/wilerpk2.ascii

 Then read /tmp/wilerpk*.ascii files: they are ASCII copies of the current binary-format package configuration that is currently running.

(You may have existing package configuration files in /etc/cmcluster/wilerpk*/ directories, but those may or may not be up to date. It is better to use cmgetconf to get the actual running configuration.)

Since your version of Serviceguard is older than A.11.18, it will be using the legacy package configuration syntax. Look for keyword LOCAL_LAN_FAILOVER_ALLOWED (or the older keyword, NET_SWITCHING_ENABLED). If it is set to NO, it prevents package IP failover. The default value when creating a new package is YES.

If it is set to NO and you want to change it to YES, edit the .ascii file, then use the cmapplyconf command to apply the change. If the change cannot be applied without shutting down the package, the cmapplyconf command will say so.

Example for wilerpk1.ascii:

cmapplyconf -P /tmp/wilerpk1.ascii

 When the cmapplyconf command is successfully completed, the .ascii files will not be needed any more by Serviceguard. But you may want to archive them, e.g. to allow reconstruction of cluster and its packages in a Disaster Recovery situation. If you keep the .ascii files, make sure that you replace all old copies of the .ascii file on each cluster node with the updated files.

[Dan Ryan_4]

Hello and thanks for your input.

Per your recommendation I've verified LOCAL_LAN_FAILOVER_ALLOWED      YES  is set in both packages.

The problem is not that the LOCAL_LAN_FAILOVER isn't happening, it just not failing over enough.

When we disable the network switch port servicing the lan0 wilerpa node  IP moves from lan0 to the standby LAN2.

However the package IP does not get moved from lan0:1 to lan2:1.

I'm wondering if the Cluster configuration need to be told about the Virtail NIC (lan0:1) and the presumed standby lan2:1?

AS configured  by the /etc/rc.config.d/netconf file the following NICs are defined:

wilerpa:# ifconfig lan0
lan0: flags=1843<UP,BROADCAST,RUNNING,MULTICAST,CKO>
        inet 10.201.209.25 netmask ffffff00 broadcast 10.201.209.255   ===> wilerpa (node)

wilerpa:# ifconfig lan0:1
lan0:1: flags=1843<UP,BROADCAST,RUNNING,MULTICAST,CKO>    
        inet 10.201.209.23 netmask ffffff00 broadcast 10.201.209.255 ===> sappd3 (package wilerpk1)

wilerpa:# ifconfig lan2
lan2: flags=842<BROADCAST,RUNNING,MULTICAST>
        inet 0.0.0.0 netmask 0      ===> Standby

wilerpa:# ifconfig lan4
lan4: flags=1843<UP,BROADCAST,RUNNING,MULTICAST,CKO>
        inet 192.168.100.1 netmask fffffff8 broadcast 192.168.100.7  ===> dedicated HB

When we disable the Switch port servicing lan0 the wilerpa node IP moves from lan0 to the standby LAN2.

However the package IP does not get moved from lan0:1 to lan2:1.

So we can get to the IP that failed over from lan0 to Lan2 but cannot get to the IP that is on lan0:1

I don't have any statements defining lan2:1 in the system's /etc/rc.config.d/netconf configuration file.

Nor do I have any statements defining NETWORK-INTERFACE lan0:1  or  NETWORK_INTERFACE lan2:1
in the Cluster configuration file.

I'm wondering if the Cluster configuration need to be told about the Virtail NIC (lan0:1) and the presumed standby lan2:1? 

I would have to add the lan2:1 configuration to the systems netconf also.

Summary I can run either package on either node, the LAN0 to LAN2 local failover is working but what to do about the Lan0:1 to Lan2:1 failover ?

[melvyn burnard]

You should NOT enter ANY package ip configuration data into the rc.config.d/netconf file. Serviceguard handles this.

It appears you ar eusing Legacy package methodology, so there is a package configuration file and a package control script.

Do you have the subnet for the package IP address specified in th econfiguration file?

Also, do you have Serviceguard patches installed? Use the "what " command:

# what /usr/lbin/cmcld |grep PHSS

[Dan Ryan_4]

Hello and thank you for this input.

I came to the same conclusion this weekend and am scheduling another round of failover testing, which will hopefully be sometime in early to mid August.

wilerpa:/# what /usr/lbin/cmcld |grep PHSS
         A.11.16.00 Date: 12/04/08 Patch: PHSS_37608

I've removed the package floating IP information from the /etc/rc.config/netconf

The subnet is defined in the package.conf file

So I'm hoping to have successful failover test during the next test shot.

[melvyn burnard]

could you attach the package config file, just for reference. If all is set ok then you may need to log a support call.

If the Sationary IP is failing over, then SG is doing it's work, but if the floating IP is not, that is an indication that the package may not be monitorig the LAN in question an dhence doies not know it has to fail over as well