ServiceGuard cluster with or without quorum server?

[Rob_Taylor]

Hi Everyone,
I've recently built a 6 node SG cluster for oracle databases running work load manager for the packages. The HP documentation I saw says that for six nodes or more it is recommended to have a quorum server. Anyone have experience on this aspect and whether it's truly necessary to have a quorum server?

Rob

[Steven E. Protter]

Shalom,

A six node cluster requires a quorum server.

SEP

[Wouter Jagers]

It is not -truly- necessary, but recommended indeed.

In the unlikely event that exactly 3 of your nodes get isolated from the others, you risk the formation of two identical clusters, which could then start simultaneously accessing the same disks.

As per the above, personally I would recommend having one ;-)

Cheers

[Wouter Jagers]

Oh.. I thought it was a 'really really recommended' thing, but didn't think it was technically mandatory.

But if SEP tells us otherwise, experience has tought me to trust his knowledge over mine ;-)

Cheers

[A. Clay Stephenson · Accepted solution]

Well, I have run 6-node clusters and done so without a cluster lock mechanism. A six-node cluster does not require a QS. In fact, before there were Quorum Servers, there were many clusters, including those with 6 or more nodes, which could not use a cluster lock and they managed to function. I would say that the decision to use a QS depends upon the probability of the cluster splitting into equally sized chunks. A carefully designed cluster would try to avoid that scenario. On the other hand, a little Linux box makes a dandy and cheap QS; they are very easy to set up; and they can never do harm, so nowadays I'm inclined to always use a QS.

[Ignacio Javier]

Hi:

This is part of the document,
Managing Serviceguard Version A.11.16,
Eleventh Edition
Second Printing

No Cluster Lock:

Normally, you should not configure a cluster of three or fewer nodes
without a cluster lock. In two-node clusters, a cluster lock is required.
You may consider using no cluster lock with configurations of three or
more nodes, although the decision should be affected by the fact that any
cluster may require tie-breaking. For example, if one node in a
three-node cluster is removed for maintenance, the cluster reforms as a
two-node cluster. If a tie-breaking scenario later occurs due to a node or
communication failure, the entire cluster will become unavailable.
In a cluster with four or more nodes, you may not need a cluster lock
since the chance of the cluster being split into two halves of equal size is
very small. However, be sure to configure your cluster to prevent the
failure of exactly half the nodes at one time. For example, make sure
there is no potential single point of failure such as a single LAN between
equal numbers of nodes, or that you donâ t have exactly half of the nodes
on a single power circuit

I hope it helps

Regards

[melvyn burnard]

A Quorum Server is NOT mandatory for anything over a two node cluster.
It is, however, recommend, as in the unlikely event you end up with a failure scenario that could result in a 50% tie break scenario that would result in all "surviving" nodes TOC'ing.
As I say, not required, but certainly recommended

[Wouter Jagers]

Hmz, that last one confuses me, so out of curiousity:

As per my recent (hence humble) knowledge on ServiceGuard, we would -want- half of the nodes to TOC when the cluster is split in half, and the quorum server helps achieve just that.

Without it, none of the nodes would TOC and 2 identical clusters would start, possibly destroying data integrity.

Correct me if I'm wrong ?

Cheers

[Serviceguard for Linux]

Wouter,

If a cluster splits exactly 50-50 without a QS then all nodes will TOC. This is contrary to your understanding that they all keep running. Servicegaurd is designed to prevent split brain and the associated danger of data corruption.

[melvyn burnard]

Although this document is a few years old, it may help to explain things:
http://docs.hp.com/en/B3936-90070/B3936-90070.pdf

Essentially, if exactly half of the nodes in a running cluster were to "disappear" due to a failure of some kind (NOT by having cmhaltnode run on them), then you end up in a 50% of quorum state whioch REQUIRES arbitration. If there is no Arbvitrator, such as a QS, then the remaining 50% of the nodes will TOC.

Also, if you had a "split brain" situation, with the cluster split 50-50 across two sites, and both halfs of the cluster went for the lock on the QS, only one half of the cluster would be granted this lock, the other "half" would be forced to TOC.

[A. Clay Stephenson]

... and this occurs by design because a TOC -- even of all the nodes in a cluster-- is far less evil than data corruption.

[Wouter Jagers]

Indeed. Thanks for clearing that up and making me double check. Kinda makes sense, as well.

I've only recently started working with MC/SG, so I've been reading a lot of the related documents. The extent in which the (hypothetic, I know now) disastrous effects of two identical clusters are described must have overshadowed the bit in which is stated the cluster would be unavailable. Hence my confusion.

However, I can't say I'm sorry I don't have first hand experience with the particular situation yet ;-)

Cheers,
Wout