topic Re: remove a quorum disk when adding a 3rd node to cluster in Operating System - OpenVMS

remove a quorum disk when adding a 3rd node to cluster

Paul Coviello — Wed, 13 Oct 2004 10:21:57 GMT

We are going to add an ES47 into a 2 host cluster of ES40's. VMS 7.3-2. EVA5000. We were thinking of changing modparams and sysgen to get rid of quorum disk and to keep expected votes the way it is, thinking that the ES47 would take the place of the qdisk.
Is this the right process to take?

thanks
Paul

Re: remove a quorum disk when adding a 3rd node to cluster

Åge Rønning — Wed, 13 Oct 2004 11:42:52 GMT

Yes, with a 3 node cluster most people would prefer no Q-disk and give one vote to each member. EXPECTED_VOTES should be set to the sum of all votes in the cluster(3). But you need to reboot after changing the sysgen paramaters to stop using the Q-disk.

Re: remove a quorum disk when adding a 3rd node to cluster

Paul Coviello — Wed, 13 Oct 2004 11:49:03 GMT

so would you change them, then a reboot and then bring in the 3rd member?

thanks
Paul

Re: remove a quorum disk when adding a 3rd node to cluster

Uwe Zessin — Wed, 13 Oct 2004 12:02:02 GMT

Yes, it makes sense to do it in this order - else you would have to change 3 system roots.

Re: remove a quorum disk when adding a 3rd node to cluster

Dima Bessonov — Thu, 14 Oct 2004 06:41:02 GMT

The interesting point here is, what value would you assign to EXPECTED_VOTES. Definitely _not_ 3 - otherwise you'll suspend the entire cluster every time you want to reboot a node. By default AUTOGEN will calculate it as 2 if you have three nodes with 1 vote each. Still, when you start up the entire cluster, you won't get it running until you bring at least two nodes up. To be able to work right after the first node is up, you must hard-code EXPECTED_VOTES as 1 in MODPARAMS.DAT. But then there's a risk of cluster partitioning and thus corrupting the file system on the disks. This may happen when your storage interconnect is not same as cluster interconnect (say, the first one is SCSI and the second one is Ethernet) and the cluster interconnect has failed. Imho, if your cluster has more than one path for cluster communications (say, redundant LAN adapters) this risk is almost nonexistent and you may safely use EXPECTED_VOTES=1 and have your cluster available right after the first node has completed its startup. Community, your opinions?

Re: remove a quorum disk when adding a 3rd node to cluster

Kris Clippeleyr — Thu, 14 Oct 2004 07:08:17 GMT

Dmitry,

The interesting point here is, what value would you assign to EXPECTED_VOTES. Definitely _not_ 3 - otherwise you'll suspend the entire cluster every time you want to reboot a node.

Definitely NOT true. In a cluster with 3 voting members, you can safely set EXPECTED_VOTES to 3. Even if a voting member crashes, the cluster happily continues because of the of the fact that quorum is calculated as:
(sum of all votes of all voting members + 2) / 2
In this case that means:
(3+2)/2 = 2.
So the cluster is happy if 2 voting members (one vote each) are present.

By default AUTOGEN will calculate it as 2 if you have three nodes with 1 vote each.

I'm not sure about that, maybe someone else can shed a light on this.

Still, when you start up the entire cluster, you won't get it running until you bring at least two nodes up.

You have a point there. But in a cluster with three voting members, don't you want at least 2 of them to be present, before you turn over the cluster to the end-users.
Certainly I would.

Greetz,

Kris

Re: remove a quorum disk when adding a 3rd node to cluster

Dima Bessonov — Thu, 14 Oct 2004 07:39:35 GMT

Kris,

Agreed that EXPECTED_VOTES=3 will be OK for two remaining nodes.

> But in a cluster with three voting
> members, don't you want at least 2 of
> them to be present, before you turn over
> the cluster to the end-users.
> Certainly I would.

It depends. Imho, the more time my cluster is available, the better for users. To me, performance is secondary to availability. And consider another situation: you're a system manager doing an upgrade or another change. No chance to check if everything went well until _two_ nodes finished startup. Another point: some upgrades simply require to shut down all other nodes booting from the same system disk. Do you prefer to change EXPECTED_VOTES and reboot every such time?

Re: remove a quorum disk when adding a 3rd node to cluster

Uwe Zessin — Thu, 14 Oct 2004 11:49:16 GMT

If you set EXPECTED_VOTES too low and you start your nodes without realizing that they can not talk together - well, you will get a partitioned cluster (better: multiple single-node clusters) that will corrupt you disks! I ONLY set it down that low when I want to boot a single node and I *know* that all others are down. I would then boot the next node(s) with higher values of EXPECTED_VOTES.

By the way: there has been a change in behaviour: I don't know when it changed, but I know that in earlier OpenVMS releases a single node was able to form a VMScluster, but then blocked due to not enough votes for quorum. The system manager could then get that node life with a quorum recalculation request through the 'IPC>' mechanism. I have noted that with V7.3-2 a cluster is only formed when enough votes for the quorum are present. Indeed that requires changing EXPECTED_VOTES through a conversational bootstrap, but - as I already wrote - I would NOT hardcode it.

Re: remove a quorum disk when adding a 3rd node to cluster

Paul Coviello — Thu, 14 Oct 2004 12:31:35 GMT

ok now that we have gone thru that! what is the consensus on what the the votes should be?

thanks
Paul

Re: remove a quorum disk when adding a 3rd node to cluster

Uwe Zessin — Thu, 14 Oct 2004 12:37:00 GMT

For normal operation, assuming that all nodes have same importancy?

Get rid of the quorum disk und give each node VOTES=1 and EXPECTED_VOTES=3.

Re: remove a quorum disk when adding a 3rd node to cluster

Kris Clippeleyr — Fri, 15 Oct 2004 05:01:43 GMT

Paul,

I'm with Uwe.

For normal operation, assuming that all nodes have same importancy?

Get rid of the quorum disk und give each node VOTES=1 and EXPECTED_VOTES=3.

I might add that we always follow the advise in the cluster manual, i.e. set the EXPECTED_VOTES to the sum of all votes of all voting members in the cluster. Remember that the EXPECTED_VOTES parameter is only used to calculate quorum when the node boots, thereafter quorum is dynamically calculated using the formula:
((sum of all votes of all voting members)+2)/2

Greetz,

Kris.

Re: remove a quorum disk when adding a 3rd node to cluster

Martin Hoogenboom — Fri, 15 Oct 2004 05:14:30 GMT

Paul,

Considering all that has been said I'm back to my preferred situation: a three (or more) node cluster without a quorum disk.

Assuming that the quorum disk is in the SAN this is preferable anyhow.

I usually hardcode only the dominant system with EXPECTED_VOTES = 1 so that i can -in the extreme situation of a cluster shutdown- can start-up one node. (Dominant in your case would be the ES47 but could also be
the system u usually?? boot first)
All other systems have expected votes at (nr-of-nodes + 1 / 2), 2 in this case.

In the systartup you can than create some switch/logical/startup-parameter that holds the actual start of applications and logins so that in specific cases you can keep that one-node cluster to yourself.
Normal behaviour is that after a certain timeout applications are started anyway.

Note that this has to be well documented, and maybe even presented during/after startup, since it is not uncommon that VMS cluster reboots occur at a lower frequency than changes in the system management team.

Regards,
Martin Hoogenboom

Re: remove a quorum disk when adding a 3rd node to cluster

Uwe Zessin — Fri, 15 Oct 2004 08:11:36 GMT

Martin,
you can do a conversational and then set EXPECTED_VOTES=1. Again, else you risk a disk corruption when your 'dominant' system has a network communications problem.

> ll other systems have expected votes at
> (nr-of-nodes + 1 / 2), 2 in this case.

And EXPECTED_VOTES is supposed to be set to the *total* number of all votes added together - you don't use the quorum calculation. The QUORUM system parameter is long obsolete.

Re: remove a quorum disk when adding a 3rd node to cluster

Jan van den Ende — Fri, 15 Oct 2004 09:35:51 GMT

First:

I _ABSOLUTELY_ agree with Uwe!! No fooling around with EXPECTED_VOTES!!

Next: if a node is _PLANNED_ to leave the cluster (say, for maintenance), then the shutdown option REMOVE_NODE will adjust Quorum downward (but, going from 3 votes present to 2 votes present results in quorum remaining at 2 :-(
).

Then, back to the original question:

would you want to do it on-the-fly, or do you prefer cluster shutdown and restart?

The latter is already nicely covered by Uwe.
Conversational bootstrap is the way to go.

If you want to do it 'rolling', that is just as easy.

I assume each node has 1 vote, EXPECTED = 3
1. Add new system (cluster_config does the work for you, specify NO Qdsk).
- cluster now running 3 nodes, 1 Qdsk, 4 votes => Quorum=(4+2)/2 = 3
2. Dismount Qdsk.
- Quorum still 3, total votes 3. Better not to lose a node!
3. $ SET CLUSTER/EXPECTED.
- recalculates quorum: (3+2)/2 = 2
4. Modify modparams for one old node to set QDSK to "" and QDSKVOTES to 0 (zero).
5. Autogen Reboot
6. repeat 4-5 for other node.

--- and if you have specific reasons to give more weight to some node compared to others, you can do some nice tricks with numbers of votes per node. But do them "dry-swimming" as paper exercises first, and consider worst cases. Compare those to the results of equal votes. MAKE SURE UNDERSTAND THE QUORUM MECHANISM COMPLETELY BEFORE TRYING ANY STUNTS!!!!

Success.

Cheers.

Have one on me.

Jan

Success.

Re: remove a quorum disk when adding a 3rd node to cluster

Keith Parris — Mon, 18 Oct 2004 13:27:16 GMT

As was stated, the setting of EXPECTED_VOTES is the only hint a system booting (or crashing and rebooting) has in order to calculate a value for Quorum, so a hard-coded setting of EXPECTED_VOTES=1 on any system is risky.

And for safety's sake, if you ever do a conversational boot to do SET EXPECTED_VOTES 1 to boot a single node alone, also do SET WRITESYSPARMS 0 so that the setting doesn't get written to ALPHAVMSPARAMS.DAT and get retained beyond the current boot.

Re: remove a quorum disk when adding a 3rd node to cluster

Paul Coviello — Wed, 20 Oct 2004 11:41:06 GMT

I want to thank you all for your insight and since we will be installing gigabit network cards for cluster communication, we will bring down the whole cluster...

in that said the qdisk removal step is as follows...

1. assuming each node has 1 vote, EXPECTED = 3
Modify modparams for one old node to:
DISK_QUORUM="$1$DGA109" --> ""
QDSKVOTES=1 --> 0
QDSKINTERVAL=30 --> 0
2. Autogen Reboot
repeat 4-5 for other node.
3. Dismount Qdsk. Or actually it will not come up on cluster startup
$ SET CLUSTER/EXPECTED. (Side note)
- recalculates quorum: (3+2)/2 = 2
4. Add new system (cluster_config does the work for you, specify NO Qdsk).
- cluster now running 3 nodes, 3 votes => Quorum=(3+2)/2 = 3

thanks
Paul

Re: remove a quorum disk when adding a 3rd node to cluster

Uwe Zessin — Wed, 20 Oct 2004 11:45:46 GMT

Paul,
3 Votes will result in quorum value of 2. You can leave QDSKVOTES and QDSKINTERVAL as they are (looks like an old cluster - I can't check right now, but I think the new default for QDSKINTERVAL is 3 and the old one was 10).

Re: remove a quorum disk when adding a 3rd node to cluster

Keith Parris — Wed, 20 Oct 2004 15:48:09 GMT

> so that the setting doesn't get written to ALPHAVMSPARAMS.DAT <

Oops! Should have been ALPHAVMSSYS.PAR

Re: remove a quorum disk when adding a 3rd node to cluster

Cass Witkowski — Fri, 22 Oct 2004 01:29:23 GMT

Do you really want to remove the quorum disk? With three nodes you have to have two nodes up to maintain a quorum. If all three nodes have direct access to the quorum disk then you can keep the quorum disk. Set the expected_votes to 5 and give each node a vote of 1 and the quorum disk a vote of 2. This will allow one node to run even if the other two are down.

Re: remove a quorum disk when adding a 3rd node to cluster

Paul Coviello — Fri, 22 Oct 2004 06:57:34 GMT

well I guess I never asked about that scenario :-) what are the drawbacks to it? I know in 6 months we will be adding at least another ES47 into the cluster...

thanks
Paul