topic 2 node Red Hat Cluster Help in Operating System - Linux

2 node Red Hat Cluster Help

mammadshah — Sat, 23 Jan 2010 14:29:45 GMT

Hi,

Using RHEL 5.3 and HP-ILo as fencing device. i did the following for testing Apache-Cluster.

1. root#mkqdisk -c /dev/sda1 -l DB_Q
2. root#chkconfig --level 345 qdiskd on
3. root#service qdiskd start
4. root#system-config-cluster

5. Cluster Name : jkz and selected quorum disk with following options
Interval = 1
TKO = 10
votes = 1
Minimum score = 3
Device = /dev/sda1
Label = DB_Q

6. Quorum Disk Heuristic
Program = ping -c 2 10.10.10.1
Score = 1
Interval = 2

7. Add new node to cluster
Node Name = node1.example.com
Quorum votes = 1
Node Name = node2.example.com
Quorum votes = 1

8. New Fence Device
HP ILO Device
Name = ILOGB89xxxxxx
user = manage
password = manage
Hostname = 10.10.10.100

HP ILO Device
Name = ILOGB88xxxxxx
user = manage
password = manage
Hostname = 10.10.10.101

9. selected Node1 and "Manage fencing for this node"
Add New Fencing level -> Add Fencing to this Level. selected ILOGB89xxxxxx

10. selected Node2 and "Manage fencing for this node"
Add New Fencing level -> Add Fencing to this Level. selected ILOGB88xxxxxx

11. Created failover domains "failover-cluster" and selected "node1.example.com and node2.example.com" from menu, and selected "Restrict to this Failover Domain"

12. Create Resource
New Resource = Apache Server
Name = Apache HTTP Server service
Server Root = /etc/httpd
Config File = /etc/httpd/conf/httpd.conf
httpd options = /etc/rc.d/init.d/httpd

13. Create a new Resource "File system"

Name = httpd-content
File System type = ext3
Mount point = /var/www/html
device = /dev/sdb1

14. Create a new Resource "IP "

10.10.10.200

15. Create a New Service "Web-Service"
Failover Domain = failover-cluster

And selected "Add shared resource to this service"

A. Apache HTTP Server Service
B. Httpd-Content
C. IP Address (10.10.10.200)

After doing all this. when i start cman service, it hangs and error puts in /var/log/messages

root## service cman start
Starting cluster:
Loading modules... done
Mounting configfs... done
Starting ccsd... done
Starting cman... done
Starting qdiskd... done
Starting daemons... done
Starting fencing...

root#tail /var/log/messages

Jan 23 17:31:24 node1 ccsd[17103]: Cluster is not quorate. Refusing connection.
Jan 23 17:31:24 node1 ccsd[17103]: Error while processing connect: Connection refused
Jan 23 17:31:29 node1 ccsd[17103]: Cluster is not quorate. Refusing connection.
Jan 23 17:31:29 node1 ccsd[17103]: Error while processing connect: Connection refused
Jan 23 17:31:34 node1 ccsd[17103]: Cluster is not quorate. Refusing connection.
Jan 23 17:31:34 node1 ccsd[17103]: Error while processing connect: Connection refused
Jan 23 17:31:39 node1 ccsd[17103]: Cluster is not quorate. Refusing connection.
Jan 23 17:31:39 node1 ccsd[17103]: Error while processing connect: Connection refused

Re: 2 node Red Hat Cluster Help

haytham_hosny — Sat, 23 Jan 2010 18:03:04 GMT

Hi ,

please make sure for the following:

- your firewall is turned off. Since you rebooted the firewall comes back on.
chkconfig --level 345 iptables off

and what is the kernal vesrion you use?

Thanks
Haytham Hosny

Re: 2 node Red Hat Cluster Help

mammadshah — Sun, 24 Jan 2010 05:18:36 GMT

HI,

Iptables and SElinux are disabled after installation of RHEL5.3. i am using 2.6.18-128.el5xen kernel.

Re: 2 node Red Hat Cluster Help

haytham_hosny — Sun, 24 Jan 2010 07:42:41 GMT

Hi ,

please review the below thread from linux site:

https://bugzilla.redhat.com/show_bug.cgi?id=485026

Thanks
Haytham Hosny

Re: 2 node Red Hat Cluster Help

mammadshah — Sun, 24 Jan 2010 08:12:16 GMT

i have some progress on 2 node cluster. but its not working when even one nodes shutdown(by root)/ or any other failure. error received in /var/log/messages.

"ccsd[24013]: Cluster is not quorate. Refusing connection"

i want to cluster node to respond,even if i have one node alive (2 nodes are working).

thanks

Re: 2 node Red Hat Cluster Help

Matti_Kurkela — Sun, 24 Jan 2010 11:25:16 GMT

> Minimum score = 3

> 6. Quorum Disk Heuristic
> Program = ping -c 2 10.10.10.1
> Score = 1

I don't see how your qdiskd can ever reach a minimum score of 3 with only one heuristic worth 1 point.

(If you use Conga to configure your cluster, or edit the cluster configuration file manually and distribute it with the ccs_tool, you can set up multiple heuristics for qdiskd.)

> i want to cluster node to respond,even if i have one node alive (2 nodes are working).

RedHat Cluster Suite manual says:
-----
Minimum Score

The minimum score for a node to be considered "alive". If omitted or set to 0, the default function, floor((n+1)/2), is used, where n is the sum of the heuristics scores. The Minimum Score value must never exceed the sum of the heuristic scores; otherwise, the quorum disk cannot be available.
-----

My recommendation: set the "Minimum Score" to 0 to use the default minimum score function.

Each node has 1 vote, and the qdiskd service gives 1 extra vote.

If the cluster is split down the middle (i.e. all heartbeat connections lost, but the nodes themselves are fine), then the qdiskd service will give its vote to one half of the cluster only: that will be the half that can
a) ping 10.10.10.1 successfully, and
b) access the quorum disk.

1 vote from the node itself and 1 vote from qdiskd will give one node a total of 2 votes, which is more than 50% of total expected votes, enough to give the node the authority to fence the other node out.

Meanwhile, the other node is taking a vote of its own. It gets only 1 vote (the quorum disk indicates qdiskd's vote has already been cast, so qdiskd on the other node won't cast it again). So that node "knows" it's about to drop out of the cluster: it should stop any cluster services and expect to be fenced at any time.

A RedHat "Clustering and GFS" course instructor said his personal recommendation was *not* to use system-config-cluster in RHEL 5. He preferred Conga for initial cluster set-up (because of the easy and powerful GUI), and ccs_tool for production cluster configuration management (because the ccs_tool is the most simple & reliable of the cluster configuration tools).

MK