Cluster Partitioned ! Help

[mike_695]

Three node (Alpha 4100, VMS 7.2-1) cluster with the Memory Channel Interconnect, sharing the same system disk on a shared SCSI bus.
Last night one node (B) lost the power and when it came up could not join the cluster; so it proceeded and formed its own single node cluster. The other two nodes (A and C) continued existing in their own 2-node cluster.
Since node B is more important for us, I shut down nodes A and C, for the time being.

How to get the nodes A nd C to join B again ?

TIA

[Robert Brooks_1]

First, what do you have for EXPECTED_VOTES on the three systems? Is there a quorum disk, and if so, how many votes does it contribute?
A properly-configured cluster should not partition, especially in the event of a connectivity failure.

Before getting nodes A and C to join B again, you should review the guidelines for cluster configuration to ensure that this doesn't happen again.

Is the shared SCSI bus parallel SCSI? If so, and if multipath is enabled (MPDEV = 1, which it is by default), please verify that MPDEV_REMOTE is 0 (which it is by default). Remote multipath support does not exist until V7.3-1.

It is likely that you'll need to correctly set EXPECTED_VOTES on all cluster members to get your cluster back to working order.

The safest thing to do would be to shut down
all nodes in the cluster before rebooting with the corrected SYSGEN params.

-- Rob

[Robert Gezelter]

Mike,

From your description, it would appear that the votes on the different systems are not set correctly.

I would be concerned about the possibility of file corruption. How long was the cluster operating partitioned?

I would immediately examine the VOTES and QUORUM parameters before the two other systems are booted. From the posted information, it is not possible to determine precisely what the problem is.

If the details of this are such that you do not want to post them, I may be contacted privately.

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Mike,

If what you say is true, your votes and expected votes system parameters must be wrong! The purpose of votes and expected votes is to prevent partitioning.

I'd expect to see 1 vote each for A,B and C with EXPECTED_VOTES=3 on all nodes.

Please post the output of:

$ MCR SYSGEN SHOW/CLUSTER

for each node.

Also note that if you had two independent clusters having the same disks mounted, it's likely there will be some disk corruption.

[John Gillings]

mike,

look at those timestamps, you got a three way notes collision :-)

Essentially the same answer from 3 independent sources - good evidence that it's quality advice...

[mike_695]

Thank you all for the prompt replies !

At this moment, I suspect that I might have a hardware problem with the Memory Channel interconnect for node B: (there is no green light for node B on the back of the memory channel box when the node is up).

FYI, every node's VOTES is 1 and the Expected_Votes was 3.

Here is the show/cluster info now, for the single-node cluster (only node B):

VWAB1B:SYSTEM$ mc sysgen show/cluster

VAXCLUSTER 2 1 0 2 Coded-valu
EXPECTED_VOTES 1 1 1 127 Votes
VOTES 1 1 0 127 Votes
RECNXINTERVAL 20 20 1 32767 Seconds D
DISK_QUORUM " " " " " " "ZZZZ" Ascii
QDSKVOTES 1 1 0 127 Votes

Parameters in use: Active
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
QDSKINTERVAL 10 3 1 32767 Seconds
ALLOCLASS 5 0 0 255 Pure-numbe
LOCKDIRWT 0 0 0 255 Pure-numbe
CLUSTER_CREDITS 10 10 10 128 Credits
NISCS_CONV_BOOT 0 0 0 1 Boolean
NISCS_LOAD_PEA0 0 0 0 1 Boolean
NISCS_PORT_SERV 0 0 0 3 Bitmask
MSCP_LOAD 1 0 0 16384 Coded-valu
TMSCP_LOAD 0 0 0 3 Coded-valu
MSCP_SERVE_ALL 2 0 0 15 Bit-Encode
TMSCP_SERVE_ALL 0 0 0 15 Bit-Encode
MSCP_BUFFER 128 128 16 -1 Coded-valu
MSCP_CREDITS 8 8 2 128 Coded-valu
MSCP_CMD_TMO 0 0 0 2147483647 Seconds D
TAPE_ALLOCLASS 0 0 0 255 Pure-numbe
SD_ALLOCLASS 0 0 0 255 Pure-numbe
NISCS_MAX_PKTSZ 1498 1498 1080 8192 Bytes
NISCS_LAN_OVRHD 18 18 0 256 Bytes
CWCREPRC_ENABLE 1 1 0 1 Bitmask D
SERVED_IO 0 0 0 -1 BITMASK

[Kris Clippeleyr]

Mike,

> EXPECTED_VOTES 1 1 1 127 Votes
> VOTES 1 1 0 127 Votes
> DISK_QUORUM " " " " " " "ZZZZ" Ascii

As my learned colleagues said, definitely a misconfiguration reagarding the votes.
Your node B expects to see 1 voting member while it is trying to join or form a cluster. So if it cannot see the other members in due time, it will (and unfortunately has) continued on its own.
As a rule of thumb, in a 3 node cluster, give each node 1 vote and set EXPECTED_VOTES to 3 on all nodes.

Regards,
Kris (aka Qkcl)

[mike_695]

Chris,

This is the current situation in a single-node cluster.
Please refer to the previous message:

Here is the show/cluster info now, for the single-node cluster (only node B):

And yes, all three nodes used to have EXPECTED_VOTES 3 and VOTES 1.

Thanks,

[Volker Halle]

Mike,

does the cluster have another interconnect in addition to memory channel ? It should at least have a LAN interconnect as well. If this is so, BOTH interconnects must have been non-functional, otherwise node B would have discovered the other nodes via the LAN.

Node B can only have formed it's own cluster, if VOTES=1 and EXPECTED_VOTES=1 were current at the time of the boot after the power failure. Maybe something has incorrectly written those parameters to ALPHAVMSSYS.PAR in the past and those bad parameters have been used during the boot. Have a look at MODPARAMS.DAT, SETPARAMS.DAT and SYS$SYSTEM:AGEN$PARAMS.REPORT for node B. SETPARAMS.DAT would contain the system parameter values set by the last run of AUTOGEN.

Consider to run ANAL/DISK on all disks in your cluster. Note the errors and run it again. If the same errors show up with each execution of ANAL/DISK (in a running system with open files on the disks), you may be able to identify real corruptions.

Volker.

[mike_695]

Volker,

Cluster traffic is only allowed on the memory channel. Here is the info for the Ethernrt adapter (no cluster traffic here):

-- EWA Device Summary 27-NOV-2007 12:50:17 --

LSB address = 811EA200
Device state = 00000003 RUN,INITED

UCB UCB Addr Fmt Value Client State
--- -------- --- ----- ------ -----------
EWA0 811E9D80
EWA2 812A8C80 Eth 60-03 DECNET 0017 STRTN,LEN,UNIQ,STRTD
EWA3 8133BD00 Eth 08-00 IP 0015 STRTN,UNIQ,STRTD
EWA4 8133C480 Eth 08-06 ARP 0015 STRTN,UNIQ,STRTD
EWA5 8133CC00 Eth 86-DD 0015 STRTN,UNIQ,STRTD
EWA6 813487C0 Eth 00-00 0000
EWA7 8135AC80 Eth 00-00 0000
EWA8 81360300 Eth 60-04 LAT 0015 STRTN,UNIQ,STRTD


However, I still suspect a hardware problem with the memory channel (for node B).

[Volker Halle]

Mike,

it is always a wise decision, to use more than 1 cluster interconnect - as you have sadly experienced ! You can use priorities to reduce the load on the LAN with more recent versions of OpenVMS. By default, memory channel will also always be preferred over the LAN interconnect SCS channels.

You normally see the advantage of this setup, if there is more than one failure at a time (in your case: a bad memory channel and bad system parameters).

Volker.

[mike_695]

You are right Volker; I should have enabled the cluster trafic over LAN as well.

I guess I can still do that by running on Node B the cluster_config.com and selecting Option 5, correct ?

CHANGE Menu
1. Enable VWAB1B as a disk server.
2. Disable VWAB1B as a disk server.
3. Enable VWAB1B as a boot server.
4. Disable VWAB1B as a boot server.
5. Enable the LAN for cluster communications on VWAB1B.
6. Disable the LAN for cluster communications on VWAB1B.

[Willem Grooters]

...but run autogen before to ensure you have the right cluster settings (Votes=1, Expected=3). Als Volker poited out, it might be that these are not (yet) set in the datafiles consulted. If VOTES=1 and EXPECTED=3, nodeB would never be able to complete boot if the connection to other nodes is lost (Votes < (Expected/2).

[mike_695]

O.K., I understand that to also (in addition to memory channel)enable LAN for cluster traffic I need to run cluster_config.com

However, how about the other two nodes (A and C) which are currently shut down ?

I assume I could SYSGEN> USE their ALPHAVMSSYS.PAR and modify NISCS_LOAD_PEA0 to 1 ?

Is this enough to enable nodes A nd C for LAN cluster traffic ?

[Volker Halle]

Mike,

as this is a common system disk, SYS$SYSTEM:CLUSTER_AUTHORIZE.DAT should be in SYS$COMMON: - please check. Booting Node A and Node C conversationally (>>> B -fl n,1 where n is the root) should be possible.

Then at the SYSBOOT prompt, set NISCS_LOAD_PEA0=1 and also check for VOTES=1 and EXPECTED_VOTES=3. Maybe set STARTUP_P1="MIN" for the first boot into the cluster to minimize damage, if something would go wrong. If everything looks o.k., type SYSBOOT> C and the node should join the existing cluster.

Volker.

[Robert Gezelter]

Mike,

Concerning the comments about using multiple cluster interconnects: It is also important to monitor the performance of the multiple interconnects. Otherwise, the first failure will likely be silent.

An automated procedure that checks connectivity information and issues alerts is an excellent time investment.

- Bob Gezelter, http://www.rlgsc.com

[Volker Halle]

Mike,

as node B is currently running without PEDRIVER loaded, you would have to reboot that node first after enabling LAN as a cluster interconnect. Node A and C would otherwise not be able to join the cluster, if memory channel still does have a problem. But by setting VOTES=1 and EXPECTED_VOTES=3, you won't get another cluster partitioning.

Volker.

[John Gillings]

mike,

>And yes, all three nodes used to have
>EXPECTED_VOTES 3 and VOTES 1.

If this is true, it is a VERY VERY serious concern. If you can, please check the sequence of events very carefully. When and how was EXPECTED_VOTES changed to 1 on node B? (the only value which will allow a single node to form a cluster on its own).

Regardless of any hardware faults or other issues, if you are able to show any circumstances that allow a system to form a cluster on its own with VOTES=1 and EXPECTED_VOTES=3 then you must log an extremely urgent case with HP to inform them that the single most fundamental mechanism for data protection in OpenVMS clusters is broken.

The assertion is somewhere in the class of "the sun came up in the West this morning" ;-) Most of us really wouldn't want to consider the potential consequences if it were true.

[Jon Pinkley]

Mike has already showed us what the problem is. EXPECTED_VOTES is set to 1. The cluster software "ratchets up" the cluster expected votes to the maximum votes seen since the formation of the cluster. This is done to protect the cluster while it is up, but when a node reboots, it believes the saved sysgen parameter values.

Mike, can you do the following:

$ set proc/priv=cmkrnl
$ analyze/system
SDA> show cluster

You will get a display similar to the following.

VMScluster data structures
--------------------------
--- VMScluster Summary ---

Quorum Votes Quorum Disk Votes Status Summary
------ ----- ----------------- --------------
2 3 1 qf_dynvote,qf_vote,quorum

--- CSB list ---

Address Node CSID ExpVotes Votes State Status
------- ---- ---- -------- ----- ----- ------

81518280 SIGMA 0001001B 3 1 local member,qf_same,qf_watcher,qf_active
816A7900 OMEGA 0001001A 3 1 open member,qf_same,qf_watcher,qf_active
816BA540 DELTA 0001001C 3 0 open member,qf_same

If any node shows ExpVotes 1, then that node can boot into its own cluster if the connectivity to the other cluster members is lost. This is what you want to avoid. Have you done an Analyze/disk on the drives that are on the shared SCSI bus?

If the shared disks were not corrupted, you were extremely lucky.

Jon

[mike_695]

O.K., let me try to clarify the initial sequence of events some more:

Node B went down due to the power failure.
When I tried to bring it on, it would not join the cluster. It definitely had EXPECTED_VOTES set to 3.
Since node B is running a critical application here, I shut down A and C and rebooted B conversationally and changed the EXPECTED_VOTES to 1.

QUESTION: I cannot find/see CLUSTER_AUTHORIZE.DAT on teh system disk !

Is it because this is a Memory Channel cluster (i.e. no LAN cluster) ?

[Volker Halle]

Mike,

CLUSTER_AUTHORIZE.DAT will be created in SYS$COMMON:[SYSEXE], if you enable LAN as a cluster interconnect. You need to specify a cluster group number (must be unique within your local network) and a password.

Volker.

[Willem Grooters]

Oher issues may be of concern if you allow cluster traffic over the LAN - even in case of failover.

* If the network gets congested- for whatever reason - it could disrupt cluster traffic causing hickups in the cluster nodes - if not crash a system.
* Cluster traffic is NOT secured so it might break security policy.

It might therefore be a good idea to have a NIC in each system for fail-over cluster communication only. I assume SCACP has facilities for that :)

WG