Changing the number of votes for the quorum disk

[Robert Atkinson]

I would reiterate Eberhard's comment. If you do not need all of your nodes to be running, but could suffice just with any one of them, then don't bother with Cluster Votes - just set the expected to 1.

Rob.

[Carleen Nutter]

To answer a few questions raised by those who've replied.

I did reboot the cluster after changing the vote related parameters; but it was before adding the new node. I edited the new nodes modparam.dat file with the vote-related parameters, so that when I booted it to join the cluster (and goes thru it's autogen and reboot), it would have the same values (votes=1,expected_votes=7, qdskvotes=3). Tech support said I would not need to reboot the entire cluster after adding the new node. The Date Formed is
consistent with the above.

I did read the cluster manual, thoroughtly.
This is a single application cluster and I considered several voting schemas. The one implemented seemed to accomodate the requirements the best.

I'll have to take some time to consider some
of the suggestions.

But perhaps, I just need to reboot the entire cluster again; and just be careful
about the shutdown. I do not have a planned
shutdown scheduled for several months.

I'll post the results in this thread at that
time.

Thanks to all for taking the time to reply and share your knowledge.

[Rob Buxton]

Just as a different tack, when shutting down Servers you can use the options that force a recalculation of Quorum.
This would allow you to shutdown Servers so that you're still left with just one Server plus the Quorum disk.

[Eberhard Wacker]

Hi again, there was so much written, so now the main item, from my point of view, in short: you DO NEED a total CLUSTER SHUTDOWN to have a situation from which you can go further on.
Regards,
Eberhard

[Jan van den Ende]

Carleen,

maybe I am missing a critical point somewhere, but as far as I was able to conclude from the info so far,

You have a 4-node cluster,
running a single application,
you can NOT fully go down ( i.e., you need 24*365 uptime),
you want to be able to shut down SOME node(s),
you SHUT DOWN those nodes in a planned way,
system CRASHES are rare.

IF these assumptions are correct, then WHY would you complicate things by HAVING a quorum disk?
A quorum disk is only an ugly (but very functional!) trick to allow a two-node cluster to survive the CRASH of any one voter (a node or maybe the quorum disk). If you grow to 3 nodes, the 3rd node (as seen from the other 2) functions just as well (actually, better, because more responsive) as a quorum disk.

In a 4-node cluster you can (if you want to) shut down 3 nodes, and continue running the 4th. (Provided you DON'T forget to REMOVE_NODE at shutdown, and you don't start a next shutdown before the previous IS down).

As long as you have at least 3 nodes running, any node crashing will leave your cluster running.

On the other side, if you have your proposed config, (4 nodes @ 1 vote, quorum disk @ 3 votes), then, if you have the situation with 1 or 2 nodes down, THEN, if you loose (connection to) the quorum disk, you WILL loose quorum!

So:
4 nodes @ 1 vote, NO QDSK, you CAN have 1 node planned-down, AND continue if another node ( = voter ) CRASHES.
4 nodes @ 1 vote, QDSK @ 3 votes, 1 node planned-down, you WILL continue through a node ( = 1-voter ) crash, but you will HANG at a Qdsk crash, OR, if your somehow loose CONNECTION to it.

IF you get down to 2 active nodes, you stand the risk of quorum loss eigther way.

If our site could count as reference:
it started as 3-node (split over 2 sites) and since has grown to 4 nodes (2 sites, 2 nodes each). We NEVER had a quorum disk.
Our current cluster uptime approaches 7 years.

I think it would even be pretty simple to remove the quorum disk without cluster shutdown:
- dismount qdsk &
- immediately SET CLUSTER/EXPECTED
- node-by-node:
---adjust SYSGEN QDSK, QDSKVOTES, & EXPECTED_VOTES, in MODPARAMS and use AUTOGEN or via WRITE CURRENT &
--- reboot.

Think it over, and if you find any wrongs in my reasoning, please post it. It would mean that I got something to learn. :-)

Succes!!

Jan

[Uwe Zessin]

Carleen,

if a new node joins a VMScluster with VOTES, the cluster quorum will be automatically adjusted upwards if necessary. EXPECTED_VOTES, as far as I can tell, is only used when a node boots. My experience is that a node that has EXPECTED_VOTES set too high cannot join a given cluster. Applying the required quorum to the rest of the cluster would cause it to hang.

You can even boot one node and it will 'hang' with a 'quorum lost' message, but if you can get to the console prompt with control-P you can request a recalculation of cluster quorum. After that you have a single-node cluster running, although your EXPECTED_VOTES setting was higher. I have attached a little .TXT file that demonstrates the required commands. Just don't forget to enter them _fast_ when you have a hung cluster with multiple nodes - otherwise the sanity timers on the other nodes expire and the node that you are working with will crash with a CLUEXIT bugcheck.

Be careful if you try this with multiple nodes! Make sure that all nodes can communicate with each other - if not you could end up with several 1-node clusters that will happily access your disks without synchronization.

The shutdown option that Rob was talking about is REMOVE_NODE. It took DEC until VMS version 6.2 to get it working, but then I could shut down a complete cluster, node for node properly. Even the last node ran without a quorum disk.

I don't know your cluster's structure (what communication busses, shared storage, ...) looks like, but I agree with Jan that a Quorum Disk can cause a lot of pain and I try to avoid it whenever possible.

[Carleen Nutter]

Definately some things to think about. In a few months, I'll have some time to do some experimenting with the cluster.

If I decide to dispense with the quorum disk, and use REMOVE_NODE during controlled shutdowns, then if I want to boot up just
1 node, am I correct that I can't just boot 1 node without playing with the expected votes during a conversational boot?
Or I must boot at lease 2 others at the same time?

Also, can anyone confirm Jan's instructions about removing the quorum disk from the cluster?

[Uwe Zessin]

Carleen,

you can boot just one node and wait until it has formed a cluster on its own. Of course it will hang with 'quorum lost'. Then you can use the command sequence I have presented above to tell that node to recalculate the quorum.

If the node has just one vote it will set the quorum to one, too. There will be a short cluster state transition and then the node will continue booting. Of course you can use the same 'trick' with 2 or more nodes. Just make sure that both nodes have joined the same cluster - you see that from the connection manager (%CNXMAN) messages. Else, as I have already written, you will get two or more independent single-node clusters which will happily eat your disks.

There is another 'trick' if you temporarily want to boot a single node:
- boot conversational
- SYSBOOT> SET EXPECTED_VOTES 1
- SYSBOOT> SET VOTES 1
- SYSBOOT> SET WRITESYSPARAMS 0

I hope I got that last name right - it is just a flag that tells the system to write back any changes made during the conversational boot back to the system parameter file. If you set it to 0, your changes don't stay over to the next boot and you don't need to bother to undo your changes.


I have never tried to remove a quorum disk from a life cluster - I don't even know if that is possible, as that information might be re-distributed from nodes that still know there was a quorum disk.

[Carleen Nutter]

A planned power outage to the data center to repair blown fuses in the ups gave me the opportunity to shutdown my cluster and reboot it. I shutdown the cluster using the /cluster_shutdown option on each node, so I wouldn't hang the cluster because of messed up votes & quorum. Later, I was able to boot just 1 node and this time CL_QDV (cluster quorum disk votes =3). When the other 3 nodes booted, the "sho cluster" output was what I expected it to be.
CL_EXP=7, CL_QUORUM=4, CL_VOTES=5, QF_VOTES=YES, CL_QDV=3.
So it looks like the problem was resolved by a full cluster reboot.

Thanks to all who contributed to this discussion. There is alot of valuable information in this thread.

[Uwe Zessin]

Carleen,
I don't see what version of OpenVMS you are using, but with V6.2 the REMOVE_NODE option of SHUTDOWN.COM finally worked properly.

If a cluster is hanging due to a quorum loss it is usually possible to request a quorum recalculation. I don't see what hardware you are using and how it is set up, but it is usually possible to press Control-P at the serial console or toggle the HALT button to get to the console prompt. Then you put the value 12(10) in the software interrupt request register - on Alpha this is:
>>> deposit sirr c

and then:
>>> continue[return]
IPC> Q[return]
IPC> [Control-Z]

That should make the cluster working again. If I recall correctly there are also external management tools like AMDS or Available Manager that run on a different system. It is possible to request a quorum recalculation from there, too. I have never worked with them and I don't do system management so I cannot tell what their status is - can somebody else tune in?