LAN config has changed

[Greg OBarr]

The LAN configuration has changed on our failover system. I get the following output from cmquerycl:

# cmquerycl -c PROD_cluster

Error: Interface lan3 on cadb02a has an IP address on it
(168.93.136.205), but the configuration file doesn't have it.
This may be caused by a local switch orchanges in the network configuration.
Failed to gather configuration information.
---------------------------------
Viewing the cluster on the node, I get:
# cmviewcl
# cmviewcl -v -n cadb02a

NODE STATUS STATE
cadb02a up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/0 lan0
PRIMARY down 0/4/0/0 lan2
STANDBY up 0/7/0/0 lan3
---------------------------------
lan2 is not defective. I can manually configure it with ifconfig and ping the IP address.

I am trying to figure out:
1) How/why did it switch?
2) How do I switch it back?

I cannot successfully query the cluster until this is fixed.

-greg

[melvyn burnard]

firstly, which version of SG? and is it patched? do:
what /usr/lbin/cmcld

try to run the command cmscancl and review the output (this assumes you are running a version that supports this command)

[Greg OBarr]

Version info as reported by "what /usr/lbin/cmcld":

cadb01a:[/etc/cmcluster]# what /usr/lbin/cmcld
/usr/lbin/cmcld:
HP92453-02A.10.20 HP-UX SYMBOLIC DEBUGGER (END.O) $Revision: 74.03 $
Build date: Mon Oct 18 10:20:10 PDT 1999
Build id: sibld_sgops_a1109
A.11.09 Date: 08/06/1999
Daemon
Config DB
Cluster Monitor
Command Srv
CommunicationSrv
Config
Dlm
Local Comm
Network Sensor
Package Manager
Remote Comm
API
Service Sensor
Cluster LVM
Status DB
Sync
Util

See attached output from cmscancl. It's showing connections as they currently exist, but as earlier stated, the way they currently exist is not the way it is supposed to be according to the cluster.conf file. Last time I have shut down either the primary OR failover system, the result has been a system lockup.... could not even get into console.

[melvyn burnard]

Without seeing log files etc this one inot easy, but it appears that the lan2 on that node is down, as in failing.

I would suggest you get a software call logged with your local HP response Centre, and/or get the mnetwork conbnection checked out.
By the way, you should patch your nodes with PHSS_22683 to ensure you have certain critical fixes in place, and this MAY be one of the symptoms.

[Greg OBarr]

Thanks. I'll log a call, but I may have neglected to mention that I can manually configure the lan card if ifconfig (different IP, of course) and it works. Problem is, I didn't set this cluster up... I came in and replaced someone else who set it up and left. I recall someone mentioning that power had been lost to some network switches over the weekend the week before I started... I'm wondering if the switch that that lan card was plugged into went down and it failed over to the other one.... it effectively lost the network on that card and would have switched over to the failover. That being the case, how do I manually switch the network back to the lan2 interface? Does SG make changes to the /etc/rc.config.d files when a lan switchover occurs?

[Peggy Fong]

Greg,
To answer your last question, ServiceGuard never changes /etc/rc.config.d/netconf
On a system reboot, netconf (startup script) will configure the cards per the netconf script. When ServiceGuard node starts it will use its own configuration file to assign primary and standby and heartbeat lans, etc. If it detects a failure (not at linklevel) it will take the card down and bring the standby up. When it detects that it can use the primar card again ServiceGuard will down the standby and bringup the primary. As someone mentioned earlier ServiceGuard thinks the card has failed. I believe it does it's testing at the ip level. You cannot manage cards under serviceguard control (you can up and down them but it is not a good idea). You say it works physically when you configured the card up - but did you then have two cards with the same ip address?

Check your routing table on both nodes and see if there are any duplicate ip addresses - this can cause confusion for serviceguard and prevent you from querying the cluster.

HTH
Peggy

[melvyn burnard]

Hmm, if you have ifconfig'ed an ip address on a lan in SG that was not in use at the time, then this COULD be the cause.
If this has happened, then when SG tries to switch to it it will get back an EBUSY (error Busy) signal and not switch it back.
You could try :
ifconfig lan2 0.0.0.0 down
and see the result, other than that, to REALLY try and test the connections, do:
cmquerycl -v -C /tmp/testasciifile -n nodename1 -n nodename2
then read through CAREFULLY the /tmp/testasciifile to see what it tells you SG sees for configuration.


HTH

[Greg OBarr]

cmquerycl -v -C /tmp/output -n cadb01a -n cadb02a reports:



# Definition of nodes in the cluster.
# Repeat node definitions as necessary for additional nodes.

NODE_NAME cadb01a
NETWORK_INTERFACE lan0
HEARTBEAT_IP 10.0.0.1
NETWORK_INTERFACE lan2
HEARTBEAT_IP 168.93.136.207
NETWORK_INTERFACE lan3

# Warning: No volume groups were found on all nodes.
# A cluster lock device is required for clusters of only two nodes.
FIRST_CLUSTER_LOCK_PV
# List of serial device file names
# For example:
# SERIAL_DEVICE_FILE /dev/tty0p0

# Warning: There are no standby network interfaces for lan0.
# Possible standby Network Interfaces for lan2: lan3.

NODE_NAME cadb02a
NETWORK_INTERFACE lan0
HEARTBEAT_IP 10.0.0.2
NETWORK_INTERFACE lan3
HEARTBEAT_IP 168.93.136.205

################
lan2 is not even shown on the output in cadb02a, but the binary cluster config file has the same net config as cadb01a (i.e. lan2 is the HEARTBEAT_IP and lan3 is the standby). This doesn't make sense.

[Greg OBarr]

As far as I know, lan2 was originally in use when they configured the cluster. I have not brought up or down any card that was in use by SG, only brought up lan2 (with a different IP address, of course) to see if it was working and I could ping it when I brought it up. It did function correctly when I brought it up with ifconfig. Then I unplumbed and plumbed it again to remove the IP address I had manually assigned.

[Greg OBarr]

I found the following entry in the syslog.log file, so the network or card must have failed at some point.

Mar 9 13:10:00 cadb02a cmcld: Starting cluster management protocols.
Mar 9 13:10:00 cadb02a cmcld: Attempting to form a new cluster
Mar 9 13:10:00 cadb02a cmcld: lan2 failed
Mar 9 13:10:00 cadb02a cmcld: Subnet 168.93.136.0 switched from lan2 to lan3
Mar 9 13:10:00 cadb02a cmcld: lan2 switched to lan3

[melvyn burnard · Accepted solution]

Greg, all th einfo you have supplied tells me you have a physical problem on that Lan card, either the cable, or the card itself. This is evidenced by the fact that cmquerycl could not even find the card.

Hope you get it sorted.

[Greg OBarr]

Thanks for the help Melvin. You're right, it was a physical problem - the cable. I found it late yesterday afternoon and replaced the cable. This is my first in-depth experience on SG and I've learned a great deal about it from this problem alone. That's usually the way it works. I don't seem to be able to assign more points to your last response... it just says "unassigned". I want to go back and put a 10 there if you know how I can do it.

Thanks to everyone else who responded too! I'll be a guru at this shortly and will return the favor.

[melvyn burnard]

Way to go!
I am glad it is fixed for you.
Regards
Melvyn