How to config VMScluster with 6 ES40s thru FC

[Song_Charles]

Hi, all
Sorry, so later to reply,

I just came back from customer's site, and still have some questions about 6_ES40 cluster. the customer's environment was formed from 6 ES40 plus MA8000 with OpenVMS/Cluster 7.3-1. For custermer's suggestion, the cluster was re_created, not roll_upgrade. ORACLE/OPS 7.3.4 was databasse supported. Node_connection were used 100MBit E_thernet, Storage_connection were used Fabric_channel, I didn't choice the QDSK fuction, every ES40 was with 1 vote, and Expected_vote was 3 votes.

and now I doubt about:

1. How many ES40 must be boot to form the cluster? and when shutdown some ES40, how many ES40 could be left in the cluster, the cluster wasn't hung.

2. How could I improve the Node_connection to aviod the cluster partition? using the second NIC port?

B+R
Charles

[Robert Brooks_1]

Charles wrote . . .

just came back from customer's site, and still have some questions about 6_ES40 cluster.

[...]

I didn't choice the QDSK fuction, every ES40 was with 1 vote, and Expected_vote was 3 votes.

and now I doubt about:

1. How many ES40 must be boot to form the cluster? and when shutdown some ES40, how many ES40 could be left in the cluster, the cluster wasn't hung.

2. How could I improve the Node_connection to aviod the cluster partition? using the second NIC port?

----------

EXPECTED_VOTES should be the sum of the VOTES sysgen param for all six systems. Since each ES40 system contributes one vote, then EXPECTED_VOTES should equal 6. Given that, it will take 4 nodes to be "up" in order to achieve quorum.

If you correctly set the EXPECTED_VOTES param, you will not run the risk of a partitioned cluster.

If you do have multiple LAN adapters, it is often a good idea to set up a private LAN that is only used by SCS (the cluster communication protocol) and no other traffic.


-- Rob

[Song_Charles]

Robï¼

I doubt also about the EXPECTED_VOTES correctly, I must change it to 6 to aviod the cluster partition.

question:

if make the change, there are must at least 4 ES40s up to get the cluster quorum?

and,
How to set the private LAN (IE1) with SCS?

last,
May I set the difference value of param Vote, EXPECTED_VOTES on each ES40?

B+R
Charles

[Volker Halle]

If you change EXCECTED_VOTES to 6 (and you MUST do that to prevent possible partitioning), then - to initially form the cluster - at least 4 ES40s must be up.

To set up a private LAN for SCS communication, just connect the secondary network interfaces (e.g. EIB0) of all 6 systems together. OpenVMS will automatically start SCS communication on all LAN adapters it finds during boot. You can check with MC LANCP SHOW LAN_DEVICE, which network interfaces are used by SCS. MC LANCP SHOW CHANNEL will show you the point-to-point circuits between all the nodes and the channels/devices being used.

You must set the SAME correct values for VOTES (=1) and EXPECTED_VOTES (=6) on all members of the cluster. This is most important, as too low a value for EXPECTED_VOTES together with a network failure preventing total connectivity may lead to partitioning during boot.

Please note that a FC cluster is very vulnerable regarding partitioning, as your disk access path (FC) and your cluster communication path (LAN) are SEPARATE and can fail separately ! But all the disks are accessible locally, so cluster partitioning will immediately put your data in danger of uncoordinated access.

The way I describe the quorum scheme, which prevents partitioning, is this:

You can not divide an apple into more than 2 parts with more than one of them being greater than half of the apple.

As long as more than half of the cluster is up (4 out of 6 nodes in your case) and you've set EXPECTED_VOTES correctly on all of them, there is no risk of partitioning.

Volker.

[Jan van den Ende]

Charles,

both Rob and Volker gave good answers. Particularly their EXPECTED_VOTES remarks are VITAL!

Another of your questions:

if make the change, there are must at least 4 ES40s up to get the cluster quorum?


The answer is only partially "yes".
If the nodes just leave, it is true.

However, if you REQUEST a node shutdown, AND you specify the REMOVE_NODE option, THEN you have a supported way of getting to a lower number of nodes.
This option recalculates the ACTUAL quorum value bases upon the remaining nodes.
Only, when you go from eg, 2 nodes, back again to the full cluster, as soon as one node with EXPECTED_VOTES = 6 joins again, then the cluster hangs temporarily until another node(s) booting adds sufficient votes to regain QUORUM (= total of 4 in your case).

hth

Proost.

Have one on me (perhaps in May in Nashua?).

jpe

[Uwe Zessin]

""Only, when you go from eg, 2 nodes, back again to the full cluster, as soon as one node with EXPECTED_VOTES = 6 joins again, then the cluster hangs temporarily until another node(s) booting adds sufficient votes to regain QUORUM (= total of 4 in your case).""

That has not been my experience. When I (accidently) did this about 8 or 9 years ago the new node with a too-high value of EXPECTED_VOTES was not let into the cluster. However, I did not get a helpful message and the cluster was doing lots of state transitions :-(

In that case you can:
- abort the boot
- do a conversational boot
- set EXPECTED_VOTES to the desired value
- set WRITESYSPARAMS=0 to prevent an update of the parameter file
- continue with the bootstrap

[Song_Charles]

After the EXPEXCTED_VOTES changed, Is it right that I must boot at least 4 ES40s up
to get the QUORUM, and When ES40 maintenance requested(shutdown with REMOVE_NODE option), the Min Number of ES40 remained (cluster not hung)is also 4?

........
as your disk access path (FC) and your cluster communication path (LAN) are SEPARATE and can fail separately !
..........
what should I do to prevent the danger?

As all oracle backgroud processes runing on every ES40, Customer could access the data by SQL-NET thru every ES40, How could I balance the application's workload?

B+R
Charles

[Volker Halle]

To prevent cluster paritioning and un-coordinated access to your FC disks, set EXPECTED_VOTES=6 as suggested.

As Jan said, you can shutdown/remove individual nodes from the cluster with REMOVE_NODE. The shutdown will re-calculate and re-set EXPECTED_VOTES to the new number of votes after excluding the votes of the node about to be shut down. If you use REMOVE_NODE you can shut down 5 of the 6 nodes and the remaining node will still continue to run.

The tricky part will be booting the nodes into the cluster again in such a state. A new node will NOT be allowed to enter the cluster, if the new cluster config (new EXPECTED_VOTES) would cause the cluster to loose quorum. If you are down to 1 node, you should then boot at least 3 other nodes at the same time, if you don't want your cluster to hang.

I would also suggest, that you make yourself familiar with the IPC interrupt for re-calculating quorum or the Availability Manager Fix Quorum function. Those will allow you to re-calculate quorum from the remaining members in the cluster and allow you to re-vive a cluster hung due to loss of quorum.

The best tool to distribute the incoming TCPIP traffic across the 6 nodes would be TCPIP LOADBROKER.

Volker.

[Ian Miller.]

I think the effective value of expected votes and quorum will have been raised as there is now six nodes with 1 vote each. However you should change the the system parameter EXPECTED_VOTES to 6 as well.
Have a look with SHOW CLUSTER/CONT then
ADD VOTE, EXPECTED, QUORUM and see.

[Ian Miller.]

See Keith Parris presentation about quorum

http://www2.openvms.org/kparris/bootcamp2005_cnxmgr.pdf