Quorum scheme comments required

[Thomas Ritter]

A three node cluster has a quorum scheme of
Each node provides 1 vote.
Quorum disk provides 3 votes.
The quorum disk is presented via an EVA.
The motivation for this configuration is to able to boot a single node.

I would have
Each node provides 1 votes. No quorum disk.
Need two nodes for a cluster. If a single node is required then with a conversational boot change expected vote to 1.
What do you think ?

[John Gillings]

Hi Thomas,

Your quorum disk only needs NODES-1 votes to be able to boot a single node, so in your case 2 votes for the quorum disk.

EXPECTED_VOTES is therefore 5, quorum is 3, so a single node plus quorum disk can form a cluster. Similarly the 3 nodes can form a cluster without quorum disk. (I'd have to look at the numbers, but I think with an odd number of nodes, it works out the same with N or N-1, but with an even number of nodes you must use N-1 otherwise you hang if you lose the quorum disk).

On a SAN allocate the smallest possible device, make it your quorum disk. Don't put any data on it. Does it even need to be mounted on each node? If not, then don't.

The other option is a quorum node. I believe it should be possible to do it with a small Alpha (or even a VAX). In theory it doesn't need to even complete STARTUP.COM, it only needs to get as far as asserting votes. No disk sharing, no network, no logins, just join the cluster and contribute votes. Technically you don't even need license PAKs, (not sure about the legal impliciations though). Unfortunately I've never been able to convince anyone to try it out, or get OpenVMS Engineering to develop a special "quorum only" package (probably only a day or two's work in DCL to develop! Not much more than finding a convenient place in STARTUP.COM to STOP).

> conversational boot change expected vote to 1.

NO! Very bad plan. That is how partitioned clusters and irrepairable data corruption happens. If you care about your data, design and strictly enforce a quorum scheme that does exactly what you need without unnecessary exposure to risks.

(and to those who claim to have been playing with EXPECTED_VOTES "for years" without any problems, good luck to you! I know of people who've been doing similar things with chainsaws and circular saws sans blade guards, without accident. I also know a few with missing digits and/or limbs! Those that remove blade guards will eventually get cut!)

[Thomas Ritter]

thanks John. Are you saying that in a three node cluster you would use a quorum disk or a quorum node ? Why not just three nodes ?

[Jon Pinkley]

Each has advantages/disadvantages.

However, for the 3 node + QDisk I would use 1 vote for each node and 2 votes for quorum disk. Expected votes 5, quorum 3. Although it really makes no difference, as the way you present it is expected votes 6, quorum 4, and 1+3 still meets the minimum quorum.

Also note that with an EVA, you can create a 1GB disk for the QDisk and can use vraid1 to protect against failure of the quorum disk. You can't use HBVS on a quorum disk (because it couldn't be trusted as a tie breaker). VMS will refuse to use a DSA device as a quorum disk, so you can't ignore the rule.

Advantages of this, cluster will operate with any single node. It will boot and will continue to work when two nodes leave.

Disadvantage, longer cluster transition times.

For the one vote each with no quorum disk. Expected votes 3, quorum 2.

Advantages: quicker transition times. Simpler without quorum disk.

Disadvantages: Can't boot single node without conversational boot. Once second node joins cluster, cluster will hang when node drops below 2. Availability manager can fix, but it requires manual intervention.
And the disadvantage that John brings up, although I am unaware of a way to avoid setting expected votes to 1 is some cases, for example when upgrading a shared system disk. You MUST be sure that the other nodes are DOWN (as in powered off) when you remove the blade guards to be safe.

Jon

[Michael Moroney]

It's generally better to avoid quorum disks, if possible. Cluster transitions with quorum disks are slower.

If you get rid of the quorum disk, set VOTES=1 and EXPECTED_VOTES=3 on all nodes. You'll need 2 nodes to be up, or, like you said, you'll have to boot with EXPECTED_VOTES=1 if you really want to have the "cluster" up with one node.

If you keep the quorum disk:

Giving it 3 votes makes it a single point of failure. No disk, no cluster, at least without rebooting with a lower EXPECTED_VOTES.

If you give it 2 votes and each node has VOTES=1 and EXPECTED_VOTES=5, you can have the cluster available if you have either 1 node plus the quorum disk, or if you have all 3 nodes available without (or with) the quorum disk.

[Jon Pinkley]

RE:"Are you saying that in a three node cluster you would use a quorum disk or a quorum node ? Why not just three nodes ?"

If you have a quorum node with its own system disk, you could give it 2 votes and never shut it down. Then you have the expected votes 5, quorum 3 cluster. So you can have the quorum node plus any one additional node, or you can have all three "production" nodes and not quorum node (for when you have to reboot the qurum node).

Jon

[John Gillings]

>Are you saying that in a three node
>cluster you would use a quorum disk or a
>quorum node ?

Thomas,

If you want to be able to boot with single node you need a Qdisk or Qnode. The node solution is potentially better, but technically that makes it a 4 node cluster, and you have to avoid the tempation to do ANYTHING on the quorum node. As soon as you have any activity on the quorum node, it becomes a critical component.

I think HP will also expect you to pay for licenses, even if you don't do any work on it. The votes will always be 1 for each "working" node and N-1 for quorum, in your case, N=3, so expected votes 5.

Ideally I'd like OpenVMS engineering to create a quorum device, just a little dongle to put on your network. Say a box with 4 ethernet connections and a micro OS. Give it a node name, cluster number & password, number of votes and expected votes. In theory I suppose it could maintain quorum for multiple clusters.

In the mean time, the cheapest and most practical is a quorum disk. As Jon suggests, a tiny virtual unit with some kind of controller based redundancy. Again avoid the tempation to store anything on it, and don't bother to back it up.

[Michael Moroney]

You can increase the cluster transition speed when using a quorum disk by decreasing the value of QDSKINTERVAL at the expense of more I/Os to the disk. Default is 3 seconds.


Re: "Ideally I'd like OpenVMS engineering to create a quorum device, just a little dongle to put on your network. Say a box with 4 ethernet connections and a micro OS. Give it a node name, cluster number & password, number of votes and expected votes. In theory I suppose it could maintain quorum for multiple clusters."

I was actually thinking the same thing: What would it take to write a minimal program that would run on some specialized hardware that would speak the minimum SCS to contribute a vote or votes to a lan cluster. However, running it on anything other than some specialized dedicated bit of hardware opens you to a point of failure. My first thought was a little program on a Linux box.

[John Gillings]

Michael,

>What would it take to write a minimal
>program that would run on some specialized
>hardware that would speak the minimum SCS
>to contribute a vote or votes to a lan
>cluster.

Simplest (and cheapest?). Take a DS10 (2 network ports built in), add extra network adapters if required (there are 4 PCI slots), but other than that you should be able to get away with a minimal configuration. Install OpenVMS on a single internal drive. Configure as a cluster node with quorum votes, cluster group, cluster password. Turn off disk sharing, don't mount any disks, set interactive logins to 0. Don't install DECwindows or TCPIP or DECnet.

Edit SYCONFIG.COM (earliest user modifiable procedure?) add the line:

$ SET PROCESS/SUSPEND/ID=0

(or maybe just STOP/ID=0?)

By that time the node has joined the cluster, but we haven't configured any hardware. We just freeze or abort the startup procedure at that point and let the cluster interrupt services handle everything from there on. Note that it should NOT have sufficient votes to boot by itself (otherwise it would be a single point of failure).

Now just plug in the various network cables and boot your cluster. You'll need at least one other node to form a cluster.

I've never had the time or resources to test this, but I believe this box will behave as a quorum node, without the need for any licenses. Since you're effectively not running the operating system, I think HP would be hard pressed to argue that you need to pay for any licenses.

We once had a MicroVAX II hacked in a similar way to behave as a remote tape drive station. It had a number of TK drives, served to the cluster, but the node didn't boot far enough to look like a full cluster node.

[Willem Grooters]

DS10 - just for quorum? This might be feasable for a site with bigger (amounts of) iron, but I would love to have one ACTIVE in my environment ;). Environmentally, it's a bad idea - using a disk uses far less energy.