cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster Lock Disk

 
SOLVED
Go to solution
rooble
Occasional Advisor

Cluster Lock Disk

HI experts,

Need your commnets on this.

When a node obtains the cluster lock, this means node has activated the cluster lock vg.

I have two node cluster A, B. node A got activated cluster lock vg i.e A obtains cluster lock disk and start to form cluster.

Suppose if anyhow commincation between these two nodes broken. Then the node that already having the clustre lock disk will form the cluster.



please correct me if I am wrong.

Regards.

Rahul
11 REPLIES
Matti_Kurkela
Honored Contributor

Re: Cluster Lock Disk

The cluster lock is updated each time the the cluster membership changes, i.e. whenever nodes join and leave. The cluster lock records the full list of cluster member nodes.

When the cluster is started and both nodes are running, then both nodes are listed in the lock as lock applicants.

Then the communication breaks.

Node A thinks: Uh oh, node B has vanished, I must form a new cluster myself.

Node B thinks: Uh oh, node A has vanished, I must form a new cluster myself.

The old cluster lock says both A and B are members of the cluster. This is no longer true, so the lock must be updated.

Both nodes will be attempting to update the cluster lock. Each node will be updating the lock so that it will list one node only: either A or B.

Let's assume that node A manages to update the lock first. After the update, the lock will list node A only. (Both nodes are on equal footing here: the node that manages to update the lock first wins.)

Node A sees its cluster lock update was successful and knows it can continue.

Node B sees its cluster lock update attempt happened too late: node A has already claimed the lock. Node B knows it must now stop all cluster activities immediately because node A will be taking over any packages that were running on node B. To do it as fast as possible, node B intentionally crashes itself.

After node B reboots after the crash, if the communication is still broken, it cannot rejoin the cluster (the automatic "cmrunnode" at boot time will time out and fail). After the communication is fixed, the sysadmin must run "cmrunnode" on node B to make it rejoin the cluster.

MK
MK
Vivek_Pendse
Valued Contributor

Re: Cluster Lock Disk

Hi,

Cluster lock disk is always shared across all (2 or 3) cluster nodes.

When you are starting the cluster, then all nodes of cluster check the connectivity to lock disk.

Now, suppose one of the server has got some hardware issues; then nodes will lose the HB connectivity; then whichever server takes the ownership or approaches the cluster lock disk first; will form the single node cluster.


More info:
Serviceguard periodically checks the health of the lock disk or LUN and writes messages
to the syslog file if the device fails the health check. This file should be monitored for early detection of lock disk problems.

Thanks,
Vivek
Michael Steele_2
Honored Contributor

Re: Cluster Lock Disk

Re: "...Then the node that already having the clustre lock disk will form the cluster..."

NodeA had the lock, it forms the cluster. The heartbeat to NodeB is broken and there is no communication. Your question is will there no be a failover?

Good question. If the cluster loses contact with the default router then it is assumed the network is down and there will be, not a node failover but a NIC failover to backup network and router.

Possibly. Configuring a 2nd default router can be tricky.

This is a big deal because in SG your heartbeat travels over that same DATA network, unless, you've got a private heartbeat network like a null modem cable directly linked between nodeA and nodeB.

However, in Veritas Cluster Server and Sun Cluster and SAP and Oracle there are polls. Polling software is started on each node and whoever answers first gets to form the cluster.

Not so in cluster lock disks for its a race between the nodes during cluster formation, both start to form one node clusters up to the point that the lock disk is aquired. Then the winner continues and the loser becomes the standby.
Support Fatherhood - Stop Family Law
Michael Steele_2
Honored Contributor

Re: Cluster Lock Disk

for more info google split brain cluster syndrome
Support Fatherhood - Stop Family Law
rooble
Occasional Advisor

Re: Cluster Lock Disk

Hi Experts,

How a node obtain cluster lock disk? by activating cluster lock vg?

regards,

Rahul
rooble
Occasional Advisor

Re: Cluster Lock Disk

Hi MK,

Below is the comment from you. I have one query with reference to your comment.

"Let's assume that node A manages to update the lock first. After the update, the lock will list node A only. (Both nodes are on equal footing here: the node that manages to update the lock first wins.)"

Could you please explain how a node update the lock(mechanism behind it)?
Vivek_Pendse
Valued Contributor

Re: Cluster Lock Disk

Hi,

As i told you before, cluster nodes periodically checks the connectivity to lockdisk & if it fails then writes the log in syslog.log.

When the nodes acquires the lock disks means, it actually writes the signature on it --> that it has been "taken" by so&so node.

Please note: Lock disk can be a package vg also. So, there is no activation (as such) happens.
Like, lock disk is part of packageA of NodeA & NodeB fails; then why the lockdisk will be activated again in package; when the package is running in NodeA. It actually writes the signature / log, that, it has been taken by NodeA.

Hope, this helps you!

Thanks,
Vivek
Michael Steele_2
Honored Contributor
Solution

Re: Cluster Lock Disk

The special area:

"...A cluster lock disk is a special area on an LVM disk located in a volume group that is shareable by all nodes in the cluster.."

"taken"

"..When a node obtains the cluster lock, this area is marked so that other nodes will recognize the lock as â taken.â .."

So the lock disk or pv, has a special area on the surface of the disk. Note that this is the lowest level and that LVM where the vg is located is the next highest 2nd level while any file system mounted to any logical volume would be the highest or third level.

There is no 3rd level or filesystem with lock disks.

For cluster formation you don't need the 2nd or LVM level where the vg is configured.

Cluster are formed by reading the special area on the surface of the lock disk. This area is either taken or free.

Think of a race condition between the nodes during cluster formation.
Support Fatherhood - Stop Family Law
rooble
Occasional Advisor

Re: Cluster Lock Disk

Hi experts,

could you please let me know how data get corrupt in two node cluster if each node try to form individual clusters?.

regards,

Rahul
P Arumugavel
Respected Contributor

Re: Cluster Lock Disk

Hi,

If a two-node cluster experiences the loss of all network connections between the nodes. This means that cluster heartbeat ceases. Each node will then try to re-form the cluster separately. This is called split-brain situation. If this were allowed to occur, it would have the potential to run the same application in two different locations and to corrupt application data. In a split-brain scenario, different incarnations of an application could end up simultaneously accessing the same disks. One incarnation might well be initiating recovery activity while the other is modifying the state of the disks. Lock disk requirement is designed to prevent a split-brain situation.

Rgds...
rooble
Occasional Advisor

Re: Cluster Lock Disk

thanks all