Single node Serviceguard cluster

[Marius Diculescu]

Hi,

I have a single node Serviceguard cluster, on HP-UX, with two NICs (lan0 and lan1) with the following configuration (actually the hostname and IP addresses are different):
# hostname
xxx

# cat /etc/hosts
127.0.0.1 localhost loopback
10.10.10.100 xxx.test.local xxx

# cat /etc/rc.config.d/netconf
HOSTNAME="xxx"
OPERATING_SYSTEM=HP-UX
LOOPBACK_ADDRESS=127.0.0.1
INTERFACE_NAME[0]=lan0
IP_ADDRESS[0]=10.10.10.100
SUBNET_MASK[0]=255.255.255.0
BROADCAST_ADDRESS[0]=10.10.10.255
INTERFACE_STATE[0]=up
DHCP_ENABLE[0]=0
INTERFACE_MODULES[0]=""

# cat /etc/cmcluster/cmclconfig.ascii
CLUSTER_NAME cluster1
NODE_NAME xxx
NETWORK_INTERFACE lan0
STATIONARY_IP 10.10.10.100
NETWORK_INTERFACE lan1
HEARTBEAT_INTERVAL 1000000
NODE_TIMEOUT 2000000
AUTO_START_TIMEOUT 600000000
NETWORK_POLLING_INTERVAL 2000000
NETWORK_FAILURE_DETECTION INONLY_OR_INOUT
MAX_CONFIGURED_PACKAGES 150

# egrep -v ^#\|^$ /etc/cmcluster/pkg1/pkg1.conf
PACKAGE_NAME pkg1
PACKAGE_TYPE FAILOVER
NODE_NAME xxx
AUTO_RUN YES
NODE_FAIL_FAST_ENABLED NO
RUN_SCRIPT /etc/cmcluster/pkg1/pkg1.cntl
HALT_SCRIPT /etc/cmcluster/pkg1/pkg1.cntl
RUN_SCRIPT_TIMEOUT NO_TIMEOUT
HALT_SCRIPT_TIMEOUT NO_TIMEOUT
SUCCESSOR_HALT_TIMEOUT NO_TIMEOUT
FAILOVER_POLICY CONFIGURED_NODE
FAILBACK_POLICY MANUAL
PRIORITY NO_PRIORITY
LOCAL_LAN_FAILOVER_ALLOWED YES
MONITORED_SUBNET 10.10.10.0
MONITORED_SUBNET_ACCESS FULL

#egrep -v ^#\|^$ /etc/cmcluster/pkg1.cntl | more
PATH=$SGSBIN:/usr/bin:/usr/sbin:/etc:/bin
VGCHANGE="vgchange -a e" # Default
CVM_ACTIVATION_CMD="vxdg -g \$DiskGroup set activation=exclusivewrite"

VXVM_DG_RETRY="NO"
DEACTIVATION_RETRY_COUNT=2
KILL_PROCESSES_ACCESSING_RAW_DEVICES="NO"
VXVOL="vxvol -g \$DiskGroup startall" # Default
FS_UMOUNT_COUNT=1
FS_MOUNT_RETRY_COUNT=0
CONCURRENT_VGCHANGE_OPERATIONS=1
ENABLE_THREADED_VGCHANGE=0
CONCURRENT_FSCK_OPERATIONS=1
CONCURRENT_MOUNT_AND_UMOUNT_OPERATIONS=1
IP[0]="10.10.10.200"
SUBNET[0]="10.10.10.0"


When I unplug the patchcord for lan0 the package IP addresses (10.10.10.200) for pkg1 doesn't move to the lan1.

Can you help me find out why ?

Thank you,
Marius

[Ivan Krastev]

Can you try with:
NETWORK_FAILURE_DETECTION INOUT

Also post some error messages - from syslog when cable is unplugged.

regards,
ivan

[Marius Diculescu]

Hi ivan,

# tail -f /var/adm/syslog/syslog.log
Oct 13 15:53:35 xxx cmnetd[29423]: Interface lan1 failed to receive, marking it down.
Oct 13 15:53:35 xxx cmnetd[29423]: lan1 failed
Oct 13 15:53:35 xxx cmnetd[29423]: Interface lan0 failed to receive, marking it down.
Oct 13 15:53:35 xxx cmnetd[29423]: lan0 failed
Oct 13 15:53:35 xxx cmnetd[29423]: Subnet 10.10.10.0 down
Oct 13 15:53:35 xxx cmcld[29415]: Subnet 10.10.10.0 in package pkg1 is down.
Oct 13 15:53:35 xxx cmcld[29415]: Failing package pkg1 on node xxx due to subnet failure.
Oct 13 15:53:36 xxx cmcld[29415]: Request from node xxx to fail package pkg1 on node xxx.
Oct 13 15:53:36 xxx cmcld[29415]: Executing '/etc/cmcluster/pkg1/pkg1.cntl stop' for package pkg1, as service PKG*63233.
Oct 13 15:53:36 xxx syslog: cmmodnet -r -i 10.10.10.200 10.10.10.0
Oct 13 15:53:36 xxx cmcld[29415]: Service PKG*63233 terminated due to an exit(0).
Oct 13 15:53:36 xxx cmcld[29415]: Halted package pkg1 on node xxx.

Failing package on node due to subnet failure


I in another tests i put NETWORK_FAILURE_DETECTION INOUT but the package IP address (10.10.10.200) for pkg1 doesn't moved to the lan1

Thank you,
Marius

[Ivan Krastev]

For the cluster both lans are down:
Oct 13 15:53:35 xxx cmnetd[29423]: Interface lan1 failed to receive, marking it down.
Oct 13 15:53:35 xxx cmnetd[29423]: lan1 failed
Oct 13 15:53:35 xxx cmnetd[29423]: Interface lan0 failed to receive, marking it down.
Oct 13 15:53:35 xxx cmnetd[29423]: lan0 failed

regards,
ivan

[Marius Diculescu]

Hi ivan,

I stopped the cluster and after that I put an IP address on lan1. Now I can ping lan1. That is lan1 has no problem.

Thank you,
Marius

[likid0]

Hy,

Try doing a linkloop to check conectivity between lan0 and lan1, so you are sure they are in the same subnet:

linkloop -i 0 lan1mac
linkloop -i 1 lan0mac

[Marius Diculescu]

Hi Orange adm,

In both cases:

Link connectivity to LAN station:
-- OK

Thank you,
Marius

[Stephen Doud]

check whether lan1 is actually a legitimate standby for lan0. Use 'cmviewconf | more' and see if lan1 is on the same bridged network as lan0 (and has not IP assigned to it).

Never assign an IP to the standby NIC, otherwise Serviceguard cannot use it as a standby NIC.

Also, only disconnect the primary NIC from the network. Disconnecting both links will cause the subnet to go down and Serviceguard will halt the package, removing the relocatable IP from any NIC referenced by the package.

[Marius Diculescu]

Hi Stephen,

# cmviewconf

Cluster information:

cluster name: cluster1
version:0
flags: 11 (no cluster lock)
heartbeat interval:1.00 (seconds)
node timeout: 2.00 (seconds)
heartbeat connection timeout: 0.00 (seconds)
io timeout extension: 0.00 (seconds)
auto start timeout: 600.00 (seconds)
network polling interval: 2.00 (seconds)
network failure detection: INOUT
first lock vg name:(not configured)
second lock vg name:(not configured)
qs host:(not configured)

Cluster Node information:

Node ID 1:
Node name: xxx

Network ID 1:
ppa: 0
old_ppa: 0
mac addr: 0xXXX
hardware path: 0/0/3/0
network interface name: lan0

IPv4 Information:
subnet: 10.10.10.0
subnet mask: 255.255.255.0
ip address: 10.10.10.100

route id: 1

IPv6 Information:

flags: 5 (Heartbeat Network)
bridged net ID: 1

Network ID 2:
ppa: 1
old_ppa: 0
mac addr: 0xXXX
hardware path: 0/1/2/0
network interface name: lan1

IPv4 Information:
subnet: 0.0.0.0
subnet mask: 0.0.0.0
ip address: 0.0.0.0

route id: 0

IPv6 Information:

flags: 2 (Non-Heartbeat Network)
bridged net ID: 1

Cluster Access Policy Information: (Not Defined)

Package information:

maximum configured packages: 150

package ID 17665:
package name: pkg1
package global flags: 5
(Package Switch Enabled)
(Package Local Switch Enabled)
(Configured Node Failover)
(Manual Failback)
package priority: (No Priority)
package run script: /etc/cmcluster/pkg1/pkg1.cntl
package run timeout: (No Timeout)
package halt script: /etc/cmcluster/pkg1/pkg1.cntl
package halt timeout: (No Timeout)
package successor halt timeout: (No Timeout)
package primary node: xxx
package subnet: 10.10.10.0
subnet access: full

package services: (Not Defined)

package dependencies: (Not Defined)

package access policies: (Not Defined)


==================================

As you can see from the above output Network ID 1 is a Heartbeat Network on IP v6 and Network ID 2 is a Non-Heartbeat Network on IP v6. Is this a problem ??? I have no IPv6 addresses defined in my /etc/rc.config.d/netconf

Network ID 1:
IPv6 Information:

flags: 5 (Heartbeat Network)
bridged net ID: 1

Network ID 2:
IPv6 Information:

flags: 2 (Non-Heartbeat Network)
bridged net ID: 1



=======================================
The standby NIC doesnâ t have an IP address as you can see from my /etc/rc.config.d/netconf
HOSTNAME="xxx"
OPERATING_SYSTEM=HP-UX
LOOPBACK_ADDRESS=127.0.0.1
INTERFACE_NAME[0]=lan0
IP_ADDRESS[0]=10.10.10.100
SUBNET_MASK[0]=255.255.255.0
BROADCAST_ADDRESS[0]=10.10.10.255
INTERFACE_STATE[0]=up
DHCP_ENABLE[0]=0
INTERFACE_MODULES[0]=""

Thank you,
Marius

[John Bigg]

From the data you show, my guess is that you are running Serviceguard 11.18, probably unpatched.

I suggest you should load the latest 11.18 patch (which is PHSS_38423 if you run HP-UX 11.23) as this contains fixes associated with lan polling and recovery which are likely to help in the case of single node clusters where you fail one of two lan cards.

[Stephen Doud]

Marius, the content of netconf is no guarantee that the root user doesn't mistakenly assign an IP to the standby NIC (lan1 in this case)

Use 'netstat -in' to verify that an IP is not assigned to lan1.

As as John stated, patch Serviceguard to harden your nodes with the most robust version available.

Also, when testing NIC/IP failover, do NOT detach lan1 (standby) from the network unless you are testing subnet failure and forced package failover. If you detach both the live NIC carrying the IP -and- the standby NIC, Serviceguard will interpret that as a down subnet and will move the package to the adoptive node if it maintains a functional subnet.

[Marius Diculescu]

Hi John,
Hi Stephen,

I am not at work this week, but Monday I will apply the patch you suggested.

Thank you,
Marius