1835978
Members
1926
Online
110088
Solutions
So you start with a 3 node cluster and pull all your connections from 1 node... Serviceguard correctly detects the failure and reforms into a 2 node cluster (the 2 remaining nodes forming a quorum of 2/3 or 66% so thats OK - they can both see each other and neither of them can see the disconnected node so they *know* that node will be toast as it will have achieved 1/3 or less than 50% and it will have TOC's itself).
Now you have a two node cluster, and thats always a special situation, as there's no-one else apart from each other to arbitrate with.... so you pull all the connections from 1 node - now neither node can talk to the other BUT they neither node can form a quorum (1/2 = 50% - not a quorum). Neither node has a quorum of less than 50%, so no TOC straight away and as neither node can talk to the other how can they *know* that the other node doesn't still have network access? (the failure could have been on a network component in-between both nodes). In this situation, simply forming a 1-node cluster just cos you can still see the network could lead to 2 1-node clusters and a split brain situation - that means corrupt data! So the only safe thing to do is use the cluster lock - in this situation that means going for the cluster lock disk. According to your posted logs, it looks like Athos got there before Porthos, and Porthos was TOC'd. If Porthos was the node with 'good' network connections that's just bad luck I'm afraid - you have simulated multiple points of failure after all.
Now on to those cluster lock disks - are you using two disk arrays in some sort of stretch cluster - there are only very special situations where you should use dual cluster lock disks - in many standard scenarios this can actually reduce availability. Review what the manual has to say about dual cluster lock disks here:
http://docs.hp.com/hpux/onlinedocs/B3936-90073/B3936-90073.html
See Chapter 3, the section on how the Cluster Manager works.
As suggested above a quorum server may work better for you - it would certainly prevent the situation you had when losing network connectivity in a 2 node cluster.
How can you be sure that a NIC is used for heartbeat only? Don't use it! By which I mean don't bind you application to that IP, and don't allow your clients to connect to it (by not advertising it in DNS or configuring it on the clients).
HTH
Duncan
I am an HPE Employee
Now you have a two node cluster, and thats always a special situation, as there's no-one else apart from each other to arbitrate with.... so you pull all the connections from 1 node - now neither node can talk to the other BUT they neither node can form a quorum (1/2 = 50% - not a quorum). Neither node has a quorum of less than 50%, so no TOC straight away and as neither node can talk to the other how can they *know* that the other node doesn't still have network access? (the failure could have been on a network component in-between both nodes). In this situation, simply forming a 1-node cluster just cos you can still see the network could lead to 2 1-node clusters and a split brain situation - that means corrupt data! So the only safe thing to do is use the cluster lock - in this situation that means going for the cluster lock disk. According to your posted logs, it looks like Athos got there before Porthos, and Porthos was TOC'd. If Porthos was the node with 'good' network connections that's just bad luck I'm afraid - you have simulated multiple points of failure after all.
Now on to those cluster lock disks - are you using two disk arrays in some sort of stretch cluster - there are only very special situations where you should use dual cluster lock disks - in many standard scenarios this can actually reduce availability. Review what the manual has to say about dual cluster lock disks here:
http://docs.hp.com/hpux/onlinedocs/B3936-90073/B3936-90073.html
See Chapter 3, the section on how the Cluster Manager works.
As suggested above a quorum server may work better for you - it would certainly prevent the situation you had when losing network connectivity in a 2 node cluster.
How can you be sure that a NIC is used for heartbeat only? Don't use it! By which I mean don't bind you application to that IP, and don't allow your clients to connect to it (by not advertising it in DNS or configuring it on the clients).
HTH
Duncan
I am an HPE Employee