topic 2 node cluster + quorum node - network breaks, what happens? in Operating System - OpenVMS

2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Thu, 31 May 2007 21:26:52 GMT

Greetings all,

There are 2 nodes (A & B), effectively primary and secondary, in isolated locations. A third node (C), the quorum node, is located in another location altogether.

Suppose the connection between A and B drops, but the connection to node C remains for both node A and B, what happens?

How do you prevent partitioning in this instance?

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Thu, 31 May 2007 21:33:07 GMT

Actually, I should add that the connection is gigabit ethernet.

Thanks,
Mark.

Re: 2 node cluster + quorum node - network breaks, what happens?

Martin Hughes — Thu, 31 May 2007 22:08:14 GMT

Either NODEA or NODEB will crash with a CLUEXIT. Assuming that they have the same number of votes then I think the one with the higher SCSSYSTEMID stays up.

Re: 2 node cluster + quorum node - network breaks, what happens?

John Gillings — Thu, 31 May 2007 23:51:12 GMT

Martin is correct, one of the nodes A or B will be forced out and will CLUEXIT. However, I don't think you can reliably determine in advance which one will be selected for all possible cases. All you can say for certain is that node C (which still has complete connectivity) will survive. Your software will need to be able to handle either node being lost.

You cannot, and should not, decide in advance that a particular node will be chosen as the survivior of a comms failure. See Keith Parris's "creeping doom" scenario to understand why.

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 00:49:12 GMT

More questions then.

As these two nodes are shadowed (A & B), and C is only a quorum, how does VMS determine that A or B should shut down if the network/link goes down between A & B?

I should have added, my apologies, that users could connect into either A or B. Perhaps obvious, perhaps not and are NOT affected by the network/link failure.

What expected_votes and vote values determine this?

Is it possible that there is a failure like this, that data could be lost in the period it takes to transition to CLUEXIT? That is, people connected to A, and people connected to B. Or is this question too vague? :-)

Thanks, Mark.

Re: 2 node cluster + quorum node - network breaks, what happens?

John Gillings — Fri, 01 Jun 2007 03:12:42 GMT

Mark,

>VMS determine that A or B should shut down
>if the network/link goes down between A & B?

Cluster rules require direct paths from each node to each other node. If a path is lost, both A and B will realise that it can't see the other one. Since both nodes can also see node C. Since each of the proposed clusters A+C and B+C have the same number of votes, there has to be a tie break, and the victim must CLUEXIT. Thems the rules.

>What expected_votes and vote values determine this?

The only sensible voting scheme for this topology is each node 1 vote. EXPECTED_VOTES is therefore 3.

You could do A=2, B=1, C=1, and therefore EXPECTED_VOTES=4, then if the A-B link breaks, B would always get kicked out because the A+C cluster has mode votes. HOWEVER, with that distribution, if you lose A, the B+C cluster is not viable, AND you're exposed to the risk of a creeping doom failure at site A.

>data could be lost

Yes incomplete transactions in flight on the "losing" node may be lost, BUT the cluster state transition logic will ensure you won't have any data corruption.

Re: 2 node cluster + quorum node - network breaks, what happens?

labadie_1 — Fri, 01 Jun 2007 03:19:13 GMT

You should have a look at Keith Parris page, at
http://www.geocities.com/keithparris/

particularly
Understanding Vaxcluster state transitions
Disk partitioning on Openvms
VmsCluster State Transitions in action
Using OpenvmsClusters for disaster tolerance

There is a lot of excellent material, I am sure I have forgotten some other documents.

Have a good reading.

Re: 2 node cluster + quorum node - network breaks, what happens?

labadie_1 — Fri, 01 Jun 2007 03:24:20 GMT

At
http://www2.openvms.org/kparris/

see "Openvms Connection Manager and the quorum scheme"
which has an example of broken Cluster.

Re: 2 node cluster + quorum node - network breaks, what happens?

Jan van den Ende — Fri, 01 Jun 2007 04:35:36 GMT

John wrote
>>>
The only sensible voting scheme for this topology is each node 1 vote. EXPECTED_VOTES is therefore 3.
<<<

Note if if you DO want an unbalanced config in which any 2 nodes can continue, then
3 + 2 + 2 with EXPECTED_VOTES=7
_IS_ a valid option.
I do not know however, if this would influence the decision of WHICH node should CLUEEXIT.
So, if the choice of the remaining cluster has either 5 or 4 votes, is the 5-vote config favored?
I am quite curious myself!

fwiw

Proost.

Have one on me.

jpe

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 05:30:54 GMT

labadie,
>At http://www2.openvms.org/kparris/
>
>see "Openvms Connection Manager and the quorum >scheme"
>which has an example of broken Cluster.

Yes, I read this, and as he states, "systems in the minority voluntarily suspend processing while systems in the majority can continue to process transactions".

The issue is that both A+C and B+C are voting equals.

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 05:47:07 GMT

John,

Thanks for your continuing help.

Mark,

>Since each of the proposed clusters A+C and >B+C have the same number of votes, there has >to be a tie break, and the victim must >CLUEXIT. Thems the rules.

Ok, I understand that now. I was getting myself worried that there would be some partitioning into 2 clusters. Ack!

>>What expected_votes and vote values >>determine this?

>The only sensible voting scheme for this >topology is each node 1 vote. EXPECTED_VOTES >is therefore 3.

So, equal votes for A+C or B+C and then let the "system" decide which one bites the dust? Ok.

>You could do A=2, B=1, C=1, and therefore >EXPECTED_VOTES=4, then if the A-B link >breaks, B would always get kicked out >because the A+C cluster has mode votes. >HOWEVER, with that distribution, if you lose >A, the B+C cluster is not viable, AND you're >exposed to the risk of a creeping doom >failure at site A.

Forgive me, why is it not viable? Does it not have 2 votes (half of 4 expected)? Or are you saying because forcing the system to make B+C the cluster could cause issues if A is still alive and chatting to C?

Re: 2 node cluster + quorum node - network breaks, what happens?

Karl Rohwedder — Fri, 01 Jun 2007 05:59:09 GMT

>Forgive me, why is it not viable? Does it not >have 2 votes (half of 4 expected)? Or are you >saying because forcing the system to make B+C >the cluster could cause issues if A is still >alive and chatting to C?

Quorum is Expectedvotes/2+1, so 4 -> 3.

regards Kalle

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 06:00:56 GMT

labadie,
> You should have a look at Keith Parris page, at http://www.geocities.com/keithparris/

Thank you, it is a very good site for information. Some of it, alas, is a bit over my head.

Merci.

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 06:02:36 GMT

Karl,

>Quorum is Expectedvotes/2+1, so 4 -> 3.

Oops, forgive me. I forgot.

Re: 2 node cluster + quorum node - network breaks, what happens?

MarkOfAus — Fri, 01 Jun 2007 06:07:58 GMT

Jan,

>Note if if you DO want an unbalanced config in which any 2 nodes can continue, then
3 + 2 + 2 with EXPECTED_VOTES=7
_IS_ a valid option.

Actually this is probably preferable if indeed it can affect the decision of which node to CLUEXIT. I would prefer A+C to up over B+C, but again, does this not lead to the "creeping doom scenario"?

Re: 2 node cluster + quorum node - network breaks, what happens?

Jan van den Ende — Fri, 01 Jun 2007 08:57:48 GMT

Mark,

actually, no; unless you set the AutoStart Action to BOOT.
If A-B connectivity breaks, then either A or B clueexits.
If on the remaining cluster (say, A + C) something ugly comes to pass, then the cluster looses another node, and goes into quotum hang.
Only to be released by human intervention. (which HAD BETTER be sufficiently instructed)

Should AUTOSTART be set to BOOT, then (say) B will try to reboot. As long as A and C see one another, while B sees C and not A, B will not be allowed to join.
In the creeping doom situation, maybe C will eventually loose connectivity to A, in which case B + C is valid. And NOW you loose all that was done on A...
Then again, if the entire A site is destroyed, so is the data at A.
_IF_ in this scenario the data is on 3-member shadow sets however, first, the members at B are disconnected. Then A goes, as do the A shadow members. So, B joins. But the ACTIVE shadow set is still at C (allbeit single member) and the DATA survives.
-- Incidentally that is why we like to be able to have two members at each site ( + one to be split off for Backup puprposes, ie, allow at least 7 members in a set... as requested again during Bootcamp.

hth.

Proost.

Have one on me.

jpe

Re: 2 node cluster + quorum node - network breaks, what happens?

Ian P — Fri, 01 Jun 2007 11:20:39 GMT

Forgive my ignorance, but surely you have protected yourself from this scenario by making A and B see each other via site C? And the same goes for the other two pairs of systems. No triangular setup like this should be able to be broken by the loss of a single link.

Ian.

Re: 2 node cluster + quorum node - network breaks, what happens?

Ian Miller. — Fri, 01 Jun 2007 12:17:29 GMT

IanP - no, each member of a cluster has to have a direct connection to each other member.
No routing.

Re: 2 node cluster + quorum node - network breaks, what happens?

Hoff — Fri, 01 Jun 2007 15:56:08 GMT

It's true that SCS cannot currently be routed. But Ian's scheme will work just fine if you have a bridged LAN configuration, and each network link traverses a different path. And assuming latency doesn't nail you.

One of the options some sites use is (for instance) FC over IP, and (in parallel) SCS over bridged LANs.

Setting Votes and Expected_Votes is in the OpenVMS FAQ (available via <>), and a write-up on the topic is also available at <>. If you do not understand the parameters and the quorum scheme based on what you've read here already and what's in the FAQ, you really need to take the time to understand it. Or to ask questions on what you don't understand.

The Quorum scheme is THE basis for clustering. Creative or incorrect settings for the values here can seriously scrozzle your data.

Stephen Hoffman
HoffmanLabs LLC

Re: 2 node cluster + quorum node - network breaks, what happens?

Andy Bustamante — Fri, 01 Jun 2007 19:09:25 GMT

To amplify on Ian's remarks, a direct connect can be switched or bridged as long as latency doesn't become an issue.

You can also add an extra pair of network adaptors and use a cross over cable between nodes A and B to maintain cluster communications. Assuming you have both systems configured for speed/duplex or auto-negotiate, your cluster traffice will use the new path without configuration. This would keep up the primary and secondary system in the event the WAN connection drops.

"There are nine and sixty ways of contructing . . . and every single one of them is right."

Andy Bustamante