System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

is it possible to configure TWO quorum disks on Redhat cluster suit?

SOLVED
Go to solution
skt_skt
Honored Contributor

is it possible to configure TWO quorum disks on Redhat cluster suit?


Recently we saw quorum lost situation and thought about adding another one. is that supported? Any one currently using two quorum disks?

openais[17753]: [CMAN ] lost contact with quorum device
openais[17753]: [CMAN ] quorum lost, blocking activity
6 REPLIES
Steven E. Protter
Exalted Contributor

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

Shalom,

This is done at the beginning of cluster configuration.

http://www.centos.org/docs/5/html/5.1/Cluster_Administration/s1-naming-cluster-CA.html

I don't see in the interface an ability to configure two quorum disks.

A quorum disk needs to be accessible to both nodes, on shared storage.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Matti_Kurkela
Honored Contributor

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

Having two quorum disks would not increase reliability, as each node would then have to maintain contact with *both* devices to have quorum.

If having one quorum disk out of two would be acceptable, it would make the following scenario possible:

On node 1:
- Lost contact with node 2
- Errors with quorum disk A, but quorum disk B is still accessible
(errors are caused by node 2 accessing quorum disk A, but this node does not know the cause)
=> conclusion: node 2 is down and I have quorum, so this node can continue and take over the services of node 2

On node 2:
- Lost contact with node 1
- Errors with quorum disk B, but quorum disk A is still accessible
(errors are caused by node 1 accessing quorum disk B, but this node does not know the cause)
=> conclusion: node 1 is down and I have quorum, so this node can continue and take over the services of node 1

Overall result:
- both conclusions are incorrect
- Cluster is now in split-brain mode
- Massive filesystem corruption on shared disks
- Data loss
- Loss of hours of business when restoring backups and recovering
- Boss is very unhappy!


Lesson learned:

When the cluster gets split in two by connectivity problems, it is important that exactly *one* half can get the quorum.

Equally important is that the other half *must not* get the impression that it has the quorum too.

Therefore the quorum disk must be indivisible.

MK
MK
Steven Schweda
Honored Contributor

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

I thought that the idea of a quorum disk was
to break a tie among an even number of
cluster members. Using two quorum disks
would be seem to be about as useful as using
none, but would be subject to more problems.

> Having two quorum disks would [...]

Among other things, it would normally change
the number of votes required to reach a
quorum.

I don't know about Red Hat's cluster
documentation, but this is all covered in
considerable detail in the VMS documentation
(and has been for the past twenty years or
so).

http://h71000.www7.hp.com/doc/82final/6318/6318pro_020.html#quorum_strat

One thing is pretty clear there:

"Each OpenVMS Cluster system can include only
one quorum disk."
loco_vikide
Frequent Advisor
Solution

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

Hello,

I was looking into setting up a second quorum disk with RHCS 5.2, but so far don't know how. I don't know if it is allowed in RHCS 5.2. Anyone knows?

I was able to do so with RHCS 4.x.

To compensate for lack of second quorum disk, I complement quorum disk setup with a heuristics logic by pinging the gateway from clustered nodes. Exmaple:


In addition, I implement a combination of persisten scsi reservation and power fencing through ILO2 and RSA2 to ensure data integrity.

Cheers,
skt_skt
Honored Contributor

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

can u give an idea on how you implemented a two quorum disks on RHCS 4.x also how is the result so far?
loco_vikide
Frequent Advisor

Re: is it possible to configure TWO quorum disks on Redhat cluster suit?

Sorry, there was a typo in my previous response. I meant RHCS 3.x not 4.x that I played with quorum disks. It has been quite a long time. Below is the URL for the doc. Check out appendix F.2 for procedures.

http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en-3/index.html

Cheers