remove a quorum disk when adding a 3rd node to cluster

[Paul Coviello]

We are going to add an ES47 into a 2 host cluster of ES40's. VMS 7.3-2. EVA5000. We were thinking of changing modparams and sysgen to get rid of quorum disk and to keep expected votes the way it is, thinking that the ES47 would take the place of the qdisk.
Is this the right process to take?

thanks
Paul

[Åge Rønning]

Yes, with a 3 node cluster most people would prefer no Q-disk and give one vote to each member. EXPECTED_VOTES should be set to the sum of all votes in the cluster(3). But you need to reboot after changing the sysgen paramaters to stop using the Q-disk.

[Paul Coviello]

so would you change them, then a reboot and then bring in the 3rd member?

thanks
Paul

[Uwe Zessin · Accepted solution]

Yes, it makes sense to do it in this order - else you would have to change 3 system roots.

[Dima Bessonov]

The interesting point here is, what value would you assign to EXPECTED_VOTES. Definitely _not_ 3 - otherwise you'll suspend the entire cluster every time you want to reboot a node. By default AUTOGEN will calculate it as 2 if you have three nodes with 1 vote each. Still, when you start up the entire cluster, you won't get it running until you bring at least two nodes up. To be able to work right after the first node is up, you must hard-code EXPECTED_VOTES as 1 in MODPARAMS.DAT. But then there's a risk of cluster partitioning and thus corrupting the file system on the disks. This may happen when your storage interconnect is not same as cluster interconnect (say, the first one is SCSI and the second one is Ethernet) and the cluster interconnect has failed. Imho, if your cluster has more than one path for cluster communications (say, redundant LAN adapters) this risk is almost nonexistent and you may safely use EXPECTED_VOTES=1 and have your cluster available right after the first node has completed its startup. Community, your opinions?

[Kris Clippeleyr]

Dmitry,


The interesting point here is, what value would you assign to EXPECTED_VOTES. Definitely _not_ 3 - otherwise you'll suspend the entire cluster every time you want to reboot a node.


Definitely NOT true. In a cluster with 3 voting members, you can safely set EXPECTED_VOTES to 3. Even if a voting member crashes, the cluster happily continues because of the of the fact that quorum is calculated as:
(sum of all votes of all voting members + 2) / 2
In this case that means:
(3+2)/2 = 2.
So the cluster is happy if 2 voting members (one vote each) are present.


By default AUTOGEN will calculate it as 2 if you have three nodes with 1 vote each.


I'm not sure about that, maybe someone else can shed a light on this.


Still, when you start up the entire cluster, you won't get it running until you bring at least two nodes up.


You have a point there. But in a cluster with three voting members, don't you want at least 2 of them to be present, before you turn over the cluster to the end-users.
Certainly I would.

Greetz,

Kris

[Dima Bessonov]

Kris,

Agreed that EXPECTED_VOTES=3 will be OK for two remaining nodes.

> But in a cluster with three voting
> members, don't you want at least 2 of
> them to be present, before you turn over
> the cluster to the end-users.
> Certainly I would.

It depends. Imho, the more time my cluster is available, the better for users. To me, performance is secondary to availability. And consider another situation: you're a system manager doing an upgrade or another change. No chance to check if everything went well until _two_ nodes finished startup. Another point: some upgrades simply require to shut down all other nodes booting from the same system disk. Do you prefer to change EXPECTED_VOTES and reboot every such time?

[Uwe Zessin]

If you set EXPECTED_VOTES too low and you start your nodes without realizing that they can not talk together - well, you will get a partitioned cluster (better: multiple single-node clusters) that will corrupt you disks! I ONLY set it down that low when I want to boot a single node and I *know* that all others are down. I would then boot the next node(s) with higher values of EXPECTED_VOTES.

By the way: there has been a change in behaviour: I don't know when it changed, but I know that in earlier OpenVMS releases a single node was able to form a VMScluster, but then blocked due to not enough votes for quorum. The system manager could then get that node life with a quorum recalculation request through the 'IPC>' mechanism. I have noted that with V7.3-2 a cluster is only formed when enough votes for the quorum are present. Indeed that requires changing EXPECTED_VOTES through a conversational bootstrap, but - as I already wrote - I would NOT hardcode it.

[Paul Coviello]

ok now that we have gone thru that! what is the consensus on what the the votes should be?

thanks
Paul

[Uwe Zessin]

For normal operation, assuming that all nodes have same importancy?

Get rid of the quorum disk und give each node VOTES=1 and EXPECTED_VOTES=3.

[Kris Clippeleyr]

Paul,

I'm with Uwe.


For normal operation, assuming that all nodes have same importancy?

Get rid of the quorum disk und give each node VOTES=1 and EXPECTED_VOTES=3.


I might add that we always follow the advise in the cluster manual, i.e. set the EXPECTED_VOTES to the sum of all votes of all voting members in the cluster. Remember that the EXPECTED_VOTES parameter is only used to calculate quorum when the node boots, thereafter quorum is dynamically calculated using the formula:
((sum of all votes of all voting members)+2)/2

Greetz,

Kris.

[Martin Hoogenboom]

Paul,

Considering all that has been said I'm back to my preferred situation: a three (or more) node cluster without a quorum disk.

Assuming that the quorum disk is in the SAN this is preferable anyhow.

I usually hardcode only the dominant system with EXPECTED_VOTES = 1 so that i can -in the extreme situation of a cluster shutdown- can start-up one node. (Dominant in your case would be the ES47 but could also be
the system u usually?? boot first)
All other systems have expected votes at (nr-of-nodes + 1 / 2), 2 in this case.

In the systartup you can than create some switch/logical/startup-parameter that holds the actual start of applications and logins so that in specific cases you can keep that one-node cluster to yourself.
Normal behaviour is that after a certain timeout applications are started anyway.

Note that this has to be well documented, and maybe even presented during/after startup, since it is not uncommon that VMS cluster reboots occur at a lower frequency than changes in the system management team.

Regards,
Martin Hoogenboom

[Uwe Zessin]

Martin,
you can do a conversational and then set EXPECTED_VOTES=1. Again, else you risk a disk corruption when your 'dominant' system has a network communications problem.

> ll other systems have expected votes at
> (nr-of-nodes + 1 / 2), 2 in this case.

And EXPECTED_VOTES is supposed to be set to the *total* number of all votes added together - you don't use the quorum calculation. The QUORUM system parameter is long obsolete.

[Jan van den Ende]

First:

I _ABSOLUTELY_ agree with Uwe!! No fooling around with EXPECTED_VOTES!!


Next: if a node is _PLANNED_ to leave the cluster (say, for maintenance), then the shutdown option REMOVE_NODE will adjust Quorum downward (but, going from 3 votes present to 2 votes present results in quorum remaining at 2 :-(
).

Then, back to the original question:

would you want to do it on-the-fly, or do you prefer cluster shutdown and restart?

The latter is already nicely covered by Uwe.
Conversational bootstrap is the way to go.

If you want to do it 'rolling', that is just as easy.

I assume each node has 1 vote, EXPECTED = 3
1. Add new system (cluster_config does the work for you, specify NO Qdsk).
- cluster now running 3 nodes, 1 Qdsk, 4 votes => Quorum=(4+2)/2 = 3
2. Dismount Qdsk.
- Quorum still 3, total votes 3. Better not to lose a node!
3. $ SET CLUSTER/EXPECTED.
- recalculates quorum: (3+2)/2 = 2
4. Modify modparams for one old node to set QDSK to "" and QDSKVOTES to 0 (zero).
5. Autogen Reboot
6. repeat 4-5 for other node.

--- and if you have specific reasons to give more weight to some node compared to others, you can do some nice tricks with numbers of votes per node. But do them "dry-swimming" as paper exercises first, and consider worst cases. Compare those to the results of equal votes. MAKE SURE UNDERSTAND THE QUORUM MECHANISM COMPLETELY BEFORE TRYING ANY STUNTS!!!!

Success.

Cheers.

Have one on me.

Jan


Success.

[Keith Parris]

As was stated, the setting of EXPECTED_VOTES is the only hint a system booting (or crashing and rebooting) has in order to calculate a value for Quorum, so a hard-coded setting of EXPECTED_VOTES=1 on any system is risky.

And for safety's sake, if you ever do a conversational boot to do SET EXPECTED_VOTES 1 to boot a single node alone, also do SET WRITESYSPARMS 0 so that the setting doesn't get written to ALPHAVMSPARAMS.DAT and get retained beyond the current boot.

[Paul Coviello]

I want to thank you all for your insight and since we will be installing gigabit network cards for cluster communication, we will bring down the whole cluster...

in that said the qdisk removal step is as follows...

1. assuming each node has 1 vote, EXPECTED = 3
Modify modparams for one old node to:
DISK_QUORUM="$1$DGA109" --> ""
QDSKVOTES=1 --> 0
QDSKINTERVAL=30 --> 0
2. Autogen Reboot
repeat 4-5 for other node.
3. Dismount Qdsk. Or actually it will not come up on cluster startup
$ SET CLUSTER/EXPECTED. (Side note)
- recalculates quorum: (3+2)/2 = 2
4. Add new system (cluster_config does the work for you, specify NO Qdsk).
- cluster now running 3 nodes, 3 votes => Quorum=(3+2)/2 = 3

thanks
Paul

[Uwe Zessin]

Paul,
3 Votes will result in quorum value of 2. You can leave QDSKVOTES and QDSKINTERVAL as they are (looks like an old cluster - I can't check right now, but I think the new default for QDSKINTERVAL is 3 and the old one was 10).

[Keith Parris]

> so that the setting doesn't get written to ALPHAVMSPARAMS.DAT <

Oops! Should have been ALPHAVMSSYS.PAR

[Cass Witkowski]

Do you really want to remove the quorum disk? With three nodes you have to have two nodes up to maintain a quorum. If all three nodes have direct access to the quorum disk then you can keep the quorum disk. Set the expected_votes to 5 and give each node a vote of 1 and the quorum disk a vote of 2. This will allow one node to run even if the other two are down.

[Paul Coviello]

well I guess I never asked about that scenario :-) what are the drawbacks to it? I know in 6 months we will be adding at least another ES47 into the cluster...

thanks
Paul

[Jan van den Ende]

Cass,


Do you really want to remove the quorum disk? With three nodes you have to have two nodes up to maintain a quorum.


That is NOT true.
Well... it IS true if you have two nodes crashing on you at the same time.....

But if you do a normal shutdown of first one node (and DO NOT forget the REMOVE_NODE option), then as soon as that node is down you can do the same to another node, and continue happily with a one-node cluster.

This same scheme only gets a little trickier if you have four (or more) nodes which you bring down to one. THEN, to get back, you will have to boot your first returning node conversational and set a lower EXPECTED_VOTES (see above) to prevent hanging until the next one boots and brings quorum.
Actually, to be exact, read this as VOTES, not nodes, in case not all nodes have equal votes...

hth

Cheers.

Have one on me.

Jan

[Paul Coviello]

Jan, so in your scenario of 4 nodes is that with or without the qdisk?

thanks
Paul
who real soon is going to have many drinks!

[Jan van den Ende]

Paul,

that is definitely WITHOUT.

In my view (not necessary everybody's) a Quorum disk is just a tiny, stupid trick, regrettably necessary if you are unfortunate enough to be really needing a cluster, but painfully restricted to two, both potentially lonely, nodes...

I am gonna join a 100 KM traffic jam now, and when I have conquered that I will join you in spirit. I have some good beers cold.

Cheers.

Have one on me.

Jan

[Uwe Zessin]

The rules about votes, expected_votes and quorum are the same whether you have a quorum disk or not. Just consider the quorum disk as a node that is almost always up - except that it enlarges you cluster transition times :-(

Jan, I hope you're doing well... I really HATE traffic jams!

[Keith Parris]

Cass's option of using a quorum disk can make the system management easier, as by giving the quorum disk one less vote than the total of the votes of all the systems, you can shut systems down one by one to a single node without having to use the REMOVE_NODE option, and yet you can survive failure of the quorum disk and continue to run, provided all the VMS nodes are up at the time.

The downside of this is that after a node leaves unexpectedly (e.g. power supply failure, cluster interconnect hardware failure, or Control-P/Halt), after the RECNXINTERVAL period has elapsed, it will take additional time (up to 4 times QDSKINTERVAL seconds) to re-validate the quorum disk's votes, and that can cause a delay in the cluster regaining quorum. With the old default value of 10 seconds for QDSKINTERVAL, that could take up to 40 seconds, which was a long time. With the new default value of 3 seconds for QDSKINTERVAL, that could now take up to 12 seconds.

And keep in mind that anytime you use the REMOVE_NODE option on SHUTDOWN to take the cluster below a majority of the potential votes, you have voluntarily created a situation where the quorum scheme cannot totally protect you against a partitioned cluster. For example, if you take the cluster down to a single node, and it continues to run, but its LAN adapter or whatever you're using for a cluster interconnect fails, then there's nothing to prevent the other 2 nodes from booting, forming a separate cluster (as 2 votes are enough to achieve quorum in this case), and trashing the shared SAN disks. (That is, there's nothing to prevent this happening other than your human intervention as a system manager to prevent someone from trying to boot the other 2 nodes at once).