How to setup Quorum Disk in existing Redhat Cluster

[Md. Minhaz Khan]

Dear Experts,

In our environment, We have a two nodes cluster for database fail-over. But we have found that no quorum disk was configured. Now we want to setup quorum disk. I have prepared an action plan to configure this. Please verify this. If any modification required please suggest me.

Step 1:
#######

Assign 5 GB LUN from SAN storage & make this LUN accessible from both nodes in the cluster

Step 2:
#######

Say "/dev/sdc" is the newly assigned 1 GB LUN

Node1# fdisk -/dev/sdc [Partitin this LUN to "83"]

Node1# fdisk -l /dev/sdc [Check this after partition]

Node1# mkqdisk -c /dev/sdc1 -l Qdisk [Create Quorum disk]

Node1# mkqdisk -L [Check newly created Quorum disk]

Node1# cp /etc/cluster/cluster.conf /etc/cluster.conf.orig

Node1# vi /etc/cluster/cluster.conf [Add below lines after ]






Node1# ccs_tool update /etc/cluster/cluster.conf [update cluster.conf among all nodes]

Step 3:
#######

Now restart the cluster in both nodes which is as follows:

1. service rgmanager stop
2. service gfs stop, we are using Red Hat GFS
3. service clvmd stop, CLVM has been used to create clustered volumes
4. service cman stop


**To start the cluster software on a member, type the following commands in this order:

1. service cman start
2. service clvmd start
3. service gfs start
4. service rgmanager start


Thanks
Minhaz

[FLQ]

Hi Minhaz,

are you using multipath?
what is your kernel version (uname -a)?


You have a typo in the label of your qdisk.

You defined it as "mkqdisk -c /dev/sdc1 -l Qdisk" but you are specifying label=â qdiskâ in the cluster.conf.

I dont know how well the cluster manages the label vs the device so I left it out of my cluster.conf.

I would reboot both nodes instead of retarting the services.

Also try to fail the ping on "172.16.10.253" on one of the nodes to force a fence from the other node.

[Md. Minhaz Khan]

# uname -a
Linux bltabscc01.banglalinkgsm.com 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

# uname -r
2.6.18-128.el5

# getconf LONG_BIT
64

For the quorum disk, It was my typo mistake. It should be qdisk.

# mkqdisk -c /dev/sdc1 -l qdisk

**Note: We are using EMC SAN storage & powerpath software in the linux host.

Thanks
Minhaz

[DeafFrog]

hi ,

>>But we have found that no quorum disk was configured

Is fencing is the solution being used ...just wanted to know why would you like to go for a quorum on a 2 node linux cluster.

[FLQ]

Hi Minhaz,

Since you are using Powerpath, you should use the MPIO path instead of /dev/sdc1 for the quorum disk.

We had to bump up the kernel version on our clusters to 2.6.18-194 in order to have multipathing work correctly on the qdisk.

As for why using qdisk, it is essentially a third node(vote) so the remaining node can form the cluster in case of split-brain.

The remaining working node can have 50%+1 of the votes to own the cluster.

[Md. Minhaz Khan]

In our present cluster configuration, we have manual fencing. Now we want to do fencing using HP ILO.

Why need quorm disk for two node cluster:
==========================================

1) if the cluster has only 2 nodes and no quorum disk, what would be a split-brain condition turns into a "fencing war" instead.

2) In 2-node clusters the quorum disk eliminates the possibility of fencing wars if the network connections between the nodes are lost. This is usually
the primary reason to use a quorum disk in a 2-node cluster.

n example of a fencing war in a 2-node cluster with no quorum disk would be:
1.) All network connections lost between nodes A and B.

2.) Node A decides B has failed and fences it. (At the same time, node B was trying to fence node A using the same rules, but by random chance, was not quite fast enough.)

3.) Since node B was succesfully fenced, node A now knows B is down for sure. Node A takes over all the cluster services; node B reboots.

4.) Since the cluster is running in the special 2-node mode, there is no quorum check and the node B can restart cluster daemons with no connection to node A. But because the state of node A is unknown to node B, there is a problem... so node B fences node A.

5.) Since node A was succesfully fenced, node B now knows A is down for sure. Node B takes over the cluster services; node A reboots.

6.) Since the cluster is running in the special 2-node mode, there is no quorum check and the node A can restart cluster daemons with no connection to node B. But because the state of node B is unknown to node A, there is a problem... so node A fences node B.

7.) (Go back to step 3.)

This cycle will go on forever until the network connections are restored or the sysadmin stops it manually. Because of the reboot cycle, the nodes can do very little useful work, and the users won't be happy.

The quorum disk can prevent this cycle from happening. It provides (at least) one extra vote to the cluster quorum voting process (which is done by the main RedHat Cluster daemons), and makes the special 2-node mode unnecessary. The quorum voting process will prevent the fenced node from starting the cluster operations until the network connections are fixed... so the step 4) won't happen.

**Note: if the above concept is wrong for configuring quorm disk for two node cluster, please correct me.

Thanks
Minhaz

[Md. Minhaz Khan]

Dear FLQ,

Please see attached command output:

Say, "/dev/emcpowera" is the pseudo name of "/dev/sdc" or "/dev/sda" or "/dev/sda" or "/dev/sdd". Do you suggest us to use "/dev/emcpowera" instead of "/dev/sdc", "/dev/sdb","/dev/sdd", "/dev/sda"

# fdisk /dev/emcpowerc

#mkqdisk -c /dev/emcpowera1 -l qdisk


**Note: we have four paths of a single LUN because of 2 HBA card.

Thanks
Minhaz

[FLQ]

Hi Minhaz,

yes I meant to use the pseudoname instead of one of the devices (/dev/sdc1).

In case you lose the path to /dev/sdc1, you will have 3 other paths to the qdisk device.

That way, you will always have a valid path to the qdisk and the cluster can be formed with one node and have 50%+1 votes. (No split-brain)