How to config VMScluster with 6 ES40s thru FC

[Song_Charles]

Hi,

My customer's applications will be migrated,
there are 6 ES40s and 1 MA8000 with 1TB disk room, they want put all the storage and servers into one VMS Cluster togather.

for VMS cluster could be support up to 96 nodes, I think it is no problem of this configuration.

My questions:
1.the customer's Environment was 2 Nodes VMScluster with V7.3-1 now, could I still use this version to configurte 6 Nodes VMScluster?

2.All ES40s were the same degree, how would I setup the VMScluster's parameters,
such as NODE_VOTE, EXPECTED_VOTE, QDSKVOTES...

3.I would assign the DISK_QUORUM to the common system boot disk (DGA0) and the disk must be the ODS-5 file system, could the disk be used as QUORUM DISK?

4.The customer's applications were run under ORACLE7/OPS, how about the performance of this VMScluster configuration?

B+R
Charles

[Robert Brooks_1 · Accepted solution]

I'm sure you will receive more detailed responses than mine . . .

A six-node cluster is not that big; you will have no trouble having six nodes connected into a fibre channel SAN. However, you should strongly consider upgrading to a supported version of OpenVMS Alpha; V7.3-1 is no longer supported and is not eligible for prior-version support.

Unless there is something a bit different about your proposed configuration, there is little-to-no reason to have a quorum disk for a six-node cluster.


-- Rob (VMS Engineering)

[Uwe Zessin]

1.) The maximum limit of 96 nodes has been established a loong time ago, but I would see if I could bring the operating system to a current, supported level.

2. Using 6 nodes, I would not use a quorum disk, give every node VOTES=1 and EXPECTED_VOTES=6

3. I would not use a quorum disk unless absolutely needed. Usually it is only used in two-node clusters. If you insist, I would choose a HSG unit with little I/O load.

[Robert Gezelter]

Charles,

One possibility that I would consider with the goal of minimizing downtime is bringing the new ES40 and the MA8000 into the current cluster, and then rolling the applications over to the new hardware. Using the dis-similar device support inherant in the latest releases of shadowing, it is even possible to move shadowsets from the old storage to the new MA8000 without even interrupting any production applications (see my presentation from HP World 2005, slides at http://www.rlgsc.com/hptechnologyforum/2005/1146.html).

Done carefully, the process of bringing the new hardware into the existing cluster, migrating the data and applications (in either order: applications first, or data first), and then removing the old equipment from the cluster, can be done with a minimum of downtime and disruption of access to users.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[comarow]

There are of course pros and cons of having a quorum disk. The biggest advantage is of course to boot any single node, or shutdown to any single node. Some people in Sans do not like a quorum disk, because if one node can communicate with the quorum disk, but their is a short gap in network connectivity, IT will become the cluster, and ALL the rest of the nodes will clue exit.

That being said, it is a rare occurance, but can happen.

Should you wish to have a quorum disk, it is not advisable to use the system disk. It is writing and reading from the quorum disk
a qdskinterval, and that is a load on the disk. Should it not get a response in time
(the system disk is busy) you will get lost connection to quorum disk.

The quorum disk should be a very lightly used disk, not a system disk.


Best of luck.
Bob

[Jan van den Ende]

Charles,

I would be a big favourer of Bob's plan, just use the existing cluster and extend it.
We did so several times already, with NO downtime.

The one thing you REALLY should do in that case, is upgrade to 7.3-2 (and latest patches) FIRST!!
(and that can also be done by rolling upgrade!).

If you have any specific questions on the upgrades, please ask.

Success!

Proost.

Have one on me.

jpe

[Phillip Thayer]

Charles,

If it were me doing this type of upgrade I would go with what Bob suggested. I have also done this type of migration and it worked very well. The first thing is to upgrade to a newer version of the OS that can be installed on the same shared system disk that you currently have. Then add in each of the ES40's one at a time until all 6 are added to the cluster. Use the SYS$STARTUP:CLUSTER_CONFIG command procedure to add nodes to the cluster because it will adjust the SYSGEN parameters as needed and ensure you don't cobble things up. Once you add a node let the system stabilize for a day or so and then add the next node. I like to do this so that I can trace problems easier. Main thing is do one change at a time so if something doesn't work properly you know what caused it. If you do more than one change at a time then you don't know which change caused the problem or if both changes combined caused the problem. Patience is a good thing when doing these types of upgrades.

Phil

[Robert Gezelter]

Charles, et al,

There is a typographical error in my post. The correct URL for my presentation is:

http://www.rlgsc.com/hptechnologyforum/2005/1146.html

The ITRC web site incorrectly generated the URL to include the punctuation at the end of the sentence.

My apologies, normally I try to catch those before hitting POST.

- Bob Gezelter, http://www.rlgsc.com

[BR764713]

Please IGNORE for the moment, the technology involved, SYSGEN parameters, quorum, voting, all that.

Look at the DESIGN of your application, what do you need from it?

Make sure you have all the components and infrastructure that will support your requirements in all the expected, and where possible the unexpected situations, and some mitigation of the worse of all possible scenarios.

Then and ONLY THEN, do you worry about the technical setup, and this should apply regardless of if you are using VMS, Trucluster, even Microsoft Cluster, VERITAS, it really is not important. You are confusing poking a few parameters into places where you need to take several steps back.

My advice here is start with your application needs, decide how that can be best provided. You have experience and possibly a little knowledge of a superlative operating system in OpenVMS, but even that can be incorrectly configured, and not provide the service you need without a good design. Consider bringing in a consultant with VMS knowledge and experience to assist you.

The OpenVMS website has some very recently published whitepapers which may be useful, www.hp.com/go/openvms

Nic Clews.

[Ian Miller.]

You are about to embark on a non-trivial upgrade project which will require careful planning if the project is to be a success with a minium of distruption. There are many variables to be considered and it sounds like you will need assistance beyond the informal help available in forums like this one. You should get in an exteral consultant who has performed this sort of upgrade before.

You certainly should take the oppourtunity to move to a supported version of VMS.

[Song_Charles]

Hi, all
Sorry, so later to reply,

I just came back from customer's site, and still have some questions about 6_ES40 cluster. the customer's environment was formed from 6 ES40 plus MA8000 with OpenVMS/Cluster 7.3-1. For custermer's suggestion, the cluster was re_created, not roll_upgrade. ORACLE/OPS 7.3.4 was databasse supported. Node_connection were used 100MBit E_thernet, Storage_connection were used Fabric_channel, I didn't choice the QDSK fuction, every ES40 was with 1 vote, and Expected_vote was 3 votes.

and now I doubt about:

1. How many ES40 must be boot to form the cluster? and when shutdown some ES40, how many ES40 could be left in the cluster, the cluster wasn't hung.

2. How could I improve the Node_connection to aviod the cluster partition? using the second NIC port?

B+R
Charles

[Robert Brooks_1]

Charles wrote . . .

just came back from customer's site, and still have some questions about 6_ES40 cluster.

[...]

I didn't choice the QDSK fuction, every ES40 was with 1 vote, and Expected_vote was 3 votes.

and now I doubt about:

1. How many ES40 must be boot to form the cluster? and when shutdown some ES40, how many ES40 could be left in the cluster, the cluster wasn't hung.

2. How could I improve the Node_connection to aviod the cluster partition? using the second NIC port?

----------

EXPECTED_VOTES should be the sum of the VOTES sysgen param for all six systems. Since each ES40 system contributes one vote, then EXPECTED_VOTES should equal 6. Given that, it will take 4 nodes to be "up" in order to achieve quorum.

If you correctly set the EXPECTED_VOTES param, you will not run the risk of a partitioned cluster.

If you do have multiple LAN adapters, it is often a good idea to set up a private LAN that is only used by SCS (the cluster communication protocol) and no other traffic.


-- Rob

[Song_Charles]

Robï¼

I doubt also about the EXPECTED_VOTES correctly, I must change it to 6 to aviod the cluster partition.

question:

if make the change, there are must at least 4 ES40s up to get the cluster quorum?

and,
How to set the private LAN (IE1) with SCS?

last,
May I set the difference value of param Vote, EXPECTED_VOTES on each ES40?

B+R
Charles

[Volker Halle]

If you change EXCECTED_VOTES to 6 (and you MUST do that to prevent possible partitioning), then - to initially form the cluster - at least 4 ES40s must be up.

To set up a private LAN for SCS communication, just connect the secondary network interfaces (e.g. EIB0) of all 6 systems together. OpenVMS will automatically start SCS communication on all LAN adapters it finds during boot. You can check with MC LANCP SHOW LAN_DEVICE, which network interfaces are used by SCS. MC LANCP SHOW CHANNEL will show you the point-to-point circuits between all the nodes and the channels/devices being used.

You must set the SAME correct values for VOTES (=1) and EXPECTED_VOTES (=6) on all members of the cluster. This is most important, as too low a value for EXPECTED_VOTES together with a network failure preventing total connectivity may lead to partitioning during boot.

Please note that a FC cluster is very vulnerable regarding partitioning, as your disk access path (FC) and your cluster communication path (LAN) are SEPARATE and can fail separately ! But all the disks are accessible locally, so cluster partitioning will immediately put your data in danger of uncoordinated access.

The way I describe the quorum scheme, which prevents partitioning, is this:

You can not divide an apple into more than 2 parts with more than one of them being greater than half of the apple.

As long as more than half of the cluster is up (4 out of 6 nodes in your case) and you've set EXPECTED_VOTES correctly on all of them, there is no risk of partitioning.

Volker.

[Jan van den Ende]

Charles,

both Rob and Volker gave good answers. Particularly their EXPECTED_VOTES remarks are VITAL!

Another of your questions:

if make the change, there are must at least 4 ES40s up to get the cluster quorum?


The answer is only partially "yes".
If the nodes just leave, it is true.

However, if you REQUEST a node shutdown, AND you specify the REMOVE_NODE option, THEN you have a supported way of getting to a lower number of nodes.
This option recalculates the ACTUAL quorum value bases upon the remaining nodes.
Only, when you go from eg, 2 nodes, back again to the full cluster, as soon as one node with EXPECTED_VOTES = 6 joins again, then the cluster hangs temporarily until another node(s) booting adds sufficient votes to regain QUORUM (= total of 4 in your case).

hth

Proost.

Have one on me (perhaps in May in Nashua?).

jpe

[Uwe Zessin]

""Only, when you go from eg, 2 nodes, back again to the full cluster, as soon as one node with EXPECTED_VOTES = 6 joins again, then the cluster hangs temporarily until another node(s) booting adds sufficient votes to regain QUORUM (= total of 4 in your case).""

That has not been my experience. When I (accidently) did this about 8 or 9 years ago the new node with a too-high value of EXPECTED_VOTES was not let into the cluster. However, I did not get a helpful message and the cluster was doing lots of state transitions :-(

In that case you can:
- abort the boot
- do a conversational boot
- set EXPECTED_VOTES to the desired value
- set WRITESYSPARAMS=0 to prevent an update of the parameter file
- continue with the bootstrap

[Song_Charles]

After the EXPEXCTED_VOTES changed, Is it right that I must boot at least 4 ES40s up
to get the QUORUM, and When ES40 maintenance requested(shutdown with REMOVE_NODE option), the Min Number of ES40 remained (cluster not hung)is also 4?

........
as your disk access path (FC) and your cluster communication path (LAN) are SEPARATE and can fail separately !
..........
what should I do to prevent the danger?

As all oracle backgroud processes runing on every ES40, Customer could access the data by SQL-NET thru every ES40, How could I balance the application's workload?

B+R
Charles

[Volker Halle]

To prevent cluster paritioning and un-coordinated access to your FC disks, set EXPECTED_VOTES=6 as suggested.

As Jan said, you can shutdown/remove individual nodes from the cluster with REMOVE_NODE. The shutdown will re-calculate and re-set EXPECTED_VOTES to the new number of votes after excluding the votes of the node about to be shut down. If you use REMOVE_NODE you can shut down 5 of the 6 nodes and the remaining node will still continue to run.

The tricky part will be booting the nodes into the cluster again in such a state. A new node will NOT be allowed to enter the cluster, if the new cluster config (new EXPECTED_VOTES) would cause the cluster to loose quorum. If you are down to 1 node, you should then boot at least 3 other nodes at the same time, if you don't want your cluster to hang.

I would also suggest, that you make yourself familiar with the IPC interrupt for re-calculating quorum or the Availability Manager Fix Quorum function. Those will allow you to re-calculate quorum from the remaining members in the cluster and allow you to re-vive a cluster hung due to loss of quorum.

The best tool to distribute the incoming TCPIP traffic across the 6 nodes would be TCPIP LOADBROKER.

Volker.

[Ian Miller.]

I think the effective value of expected votes and quorum will have been raised as there is now six nodes with 1 vote each. However you should change the the system parameter EXPECTED_VOTES to 6 as well.
Have a look with SHOW CLUSTER/CONT then
ADD VOTE, EXPECTED, QUORUM and see.

[Ian Miller.]

See Keith Parris presentation about quorum

http://www2.openvms.org/kparris/bootcamp2005_cnxmgr.pdf

[Song_Charles]

Yes,
I had assigned 1 VOTE to each ES40,
and connected LAN IEB0 togather to private
LAN SWITCH.

another question:
which ECOs would be implemented for Fiber Channel supported under VMS 7.3-1?

Thanks all for kindly advise!

Charles

[Robert Brooks_1]

which ECOs would be implemented for Fiber Channel supported under VMS 7.3-1?

----
You want the latest FIBRE_SCSI kit.

Again, I *strongly* urge you to consider upgrading to a SUPPORTED version of OpenVMS Alpha!


-- Rob

[Song_Charles]

I should consider that upgrade OS to 73-2, I will discuss with my customer.

Charles

[Song_Charles]

After UPGRADE ACS FROM V8.6S-1 TO V8,6S-13,
THE CLUSTER HAVEN'T HUNG ABOUT 2 WEEKS, I THINK THE PROBLEMS HAD BEEN SOLVED.

THANKS

CHARLES

[Song_Charles]

thanks