Rick: you have one of the classic clustering issues, and you are on the right track with respect to your question and your guess about the probable outcome.
MCSG requires a single cluster lock disk for two-node clusters, suggests it for 3- and 4-node clusters, and doesn't allow it for 5+ node clusters. The cluster lock "disk" is just any one of the Volume Groups that are under MC/SG control.
The only information used by the cluster lock "disk" is a tiny piece in the VGRA, the Volume Group Reserved Area (space which is always set aside in any LVM disk anyway). This means that the cluster lock disk doesn't take up any actual room, and also that all you have to do is designate one of the VGs under MCSG control to be the lock disk, and those bits in the VGRA get set up for cluster lock function. It is ideal to do it this way, so that you are sure the lock disk's underlying hardware (RAID LUN or actual disk drive) are working correctly.
With a geographically separated cluster ("campus cluster" or "extended campus topology"), you have exactly the problem that you have described. This is the only case where HP recommends using dual cluster lock disks, one at each site. The good news is that this will allow one site to take over all cluster functions if a site is lost. The bad news is that both sites will attempt to run all cluster packages (and succeed), if the only thing lost is the network between the two (split-brain syndrome, with apps up on both sides, all databases open twice, generally very bad news).
The best way to reduce the liklihood of this happening is to have two (or more) completely redundant networks in place, unbridged (no points of connection other than the clustered servers). This means that the wires or fiber need to be in separate places (trenches, ceilings, plenums, etc), so that no single event can drop the whole network. There is almost no way to be extreme enough about this... just like the two computer rooms (separate power sources, even different utility companies, if possible). As long as any network is still functional, split-brain syndrome is avoided, and users can still access their apps and data.
There are two other ways to avoid any possibility of split-brain syndrome, but they (naturally) have their own headaches. One is to avoid the use of cluster locks altogether (simply don't assign any), and to instead use a small HP-UX server in a third location, a third node in the cluster. HP calls this an "Arbiter Node" or "arbitrator system", and it is a fully supported configuration (specifically intended for this exact situation). The arbiter node only needs to be on the network, it has no apps, no connection to any shared storage, and no role in running any packages (none are ever assigned to it).
Its sole purpose is to provide a quorum for whichever system is still in communications with it, if a network fails. Since more than 50% of the nodes are present, quorum is established, the cluster is rebuilt as a two-node cluster, and MCSG restarts all packages as per its control scripts (normally all start on the surviving node). The reason this works is that a cluster lock is not requred for a 3-node cluster, and any two nodes can establish the quorum.
The downside is two-fold: first, a complete network failure will still cause no node to establish a quorum, and all nodes will TOC and halt. This means no split-brain syndrome, but also no automated/lights-out failover. Somebody has to manually restart one or both nodes, and convince it that it should resume cluster operations. In a lot of cases, this is the preferred outcome, compared to split-brain syndrome. The second downside is that not everybody actually has a campus, with 3 or more buildings over which they have ownership or control. A lot of people have their main site, and a co-location center or a second building, and nothing else.
For this scheme to work, a disaster in either main site cannot affect the arbiter node -- so it needs to be in a third location, or some sort of specially powered, air conditioned, and otherwise hardened space in one of the two main sites. This can get tricky and costly, as can the requirement for the arbiter node to be on all network segments in use by the cluster. But this is the optimal way around your dilemma.
The second way to avoid split-brain syndrome is much simpler, but is unacceptable for some high-uptime requirement situations: only install one cluster lock, as you are doing, and simply be OK with the fact that you have partial failover automation. If the disaster hits the site with the cluster lock disk, and the remaining site cannot establish a quorum, it will TOC. For lots of people, this is preferable to split-brain syndrom, as mentioned earlier. Manual intervention is required to get things back up and running at the functional site, and MCSG has to be convinced to come up and run as the only node. But your data is all intact and uncorrupted, and the tasks to whip MCSG into action are merely procedural, easily documented, and even scriptable (for the most part).
So, for a site with 7x24 operations, most people prefer some manual intervention to the data corruption or other issues surrounding split-brain syndrome. If there is an actual disaster, 30 to 60 minutes to get things back on-line is an unbelievable blessing and miracle, and nobody will be fussing or complaining. If there was just a network outage and things are down for an hour for manual restart, some questions might come up later, but the answers are about the costs for special schemes described above. The costs are substantial, even overwhelming. Most of the time, everybody shuts up about the outage when the cost of total and complete automation and redundancy is finally established.
Sorry for the novel, it is a complex subject. Hope it helps... and I hope you get some other answers, I don't profess to know everything about this specialized field, in which the rules change from time to time as technology or new features add additional wrinkles or solutions.
Best Regards, --bmr
We must indeed all hang together, or, most assuredly, we shall all hang separately. (Benjamin Franklin)
