VMS Cluster design issues

[Richard Allen]

We have a 3 node OpenVMS (7.3-1) cluster running Oracle. We recently bought two ES80 machines and I'm considering whats the best way to add the machines into the cluster.

The older machines are GS140, GS60 and GS60e and use a FDDI loop for the internode traffic. They also have a single ethernet NIC and a SAN card for storage.

The machines are using TCP/IP 5.3

We are aware of OpenVMS 7.3-2 but we really do not want to upgrade just yet.

The new ES80 machines have multiple NIC's and no FDDI so Im seriously considering getting rid of the FDDI loop for a pure ethernet solution.
This requires a more fault tolerant ethernet setup, especially for the 3 older nodes.

I know from reading various documentation that 7.3-2 and/or TCP/IP 5.4 have an array of new exiting features for us to use, IP failover (Hot standby ethernet NIC's. Etherchanneling ?) and so on.

So is TCP/IP 5.4 available for 7.3-1 ? Which technologies would be best for me to use with respect to the ethernet configuration ?

Any help will be greatly appreciated :)

[Martin P.J. Zinser]

Hello,

TCP/IP 5.4 is supported on 7.3-1 and 7.3-2.

See http://h71000.www7.hp.com/doc/732FINAL/6524/6524pro.HTML

for an official statement. Not all of the failover technologies might be supported though on the older version.

On a more general note, we did recently replace FDDI as the interconnect in our cluster with Gigabit ethernet cards and Cisco switches. No complains yet.

Greetings, Martin

[Mike Naime]

I would second the reccomendation on the GIG-E interconnect.

If this is not an option, then I would look into picking up a network switch that you can set the ports to 100/full instead of Auto-negotiate. Make this a dedicated "HUB" that only your VMS systems are plugged into. This can also be setup for DECNET traffic between the nodes.

On our 2-node clusters, our DBA's setup the crossover cable to be the primary Oracle communications route.

[Ian Miller.]

gigabit ethernet is the way to go. You introduce two indepenant gigabit ethernet paths for redunancy also. TCPIP V5.4 does have failover features but only for TCPIP traffic not cluster traffic. VMS V7.3-2 has features which will apply to all traffic. By introducing GBE now you can gain some benefits and and make best use of the new features in VMS V7.3-2 when you choose to upgrade.

[Keith Parris]

One option would be to add FDDI cards to the new boxes. Another would be to use something like a VN9000FX from dnpg.com which has both FDDI and Fast Ethernet ports and tie the old systems' FDDI LAN into the new systems' Ethernet LAN that way.

FDDI and Gigabit Ethernet (provided your switches really support Jumbo frames) have the advantage of being able to handle larger packets than the 1498-byte payload size that Fast Ethernet can handle. Larger packets are useful for block data transfers, which get used for things like MSCP serving (in case you're doing any of that) and lock remastering operations between nodes. For regular lock requests, which are only about 200 bytes in size and which probably represent the bulk of your inter-node SCS traffic, Fast Ethernet packet sizes are more than sufficient.

You can have a mix of interconnects in use. You can leave the FDDI in place for communications among the existing nodes, using Fast Ethernet to talk between the old and new nodes on the old nodes' one Ethernet adapter, talk among the new nodes with another Ethernet rail, and even bridge the old FDDI to the new 2nd Ethernet rail using a VN9000FX if you want redundant paths and don't want to buy more Ethernet adapters for the old nodes (although Fast Ethernet adapters are very inexpensive now).

There are two models of Gigabit Ethernet adapters, the older DEGPA and the newer DEGXA. The older adapter could run out of steam handling small packets like lock requests, so if you have high lock rates, a DE500 or DE602 could actually be faster. (You can measure lock rates with MONITOR DLOCK, MONITOR CLUSTER, SDA> LCK SHOW ACTIVE, and the LOCK_ACTV tool from the V6 Freeware CD directory [KP_LOCKTOOLS]). The newer DEGXA can handle about 3 times the number of lock requests as a Fast Ethernet adapter. But Gigabit Ethernet adapters and switches are more expensive.

VMS can readily use multiple paths at once, so you could consider using multiple Fast Ethernet rails for now and plan to buy Gigabit Ethernet as the price comes down. But if performance is critical to you today, consider buying Gigabit Ethernet now.

You may also find it useful to compare lock request latencies between the different interconnect types. The LOCKTIME.COM tool from the V6 Freeware CD [KP_CLUSTERTOOLS] can measure this as you turn various links off and on. In general Fast Ethernet should be roughly the same or perhaps a little faster than FDDI (I've measured in the area of 240 microseconds for Fast Ethernet and perhaps 270 for FDDI on a GIGAswitch), and Gigabit Ethernet will be a bit faster (200 microseconds). Links with a cross-over cable and no switch will be faster (e.g. 140 microseconds for Gigabit Ethernet on a cross-over cable). EV7 platforms tend to have significantly lower latencies even with the same LAN adapters, due to their better I/O capabilities.

As you configure the new systems, the tools SHOW_SEGMENTS.COM and SHOW_PATHS_ECS.COM from the V6 Freeware CD directory [KP_CLUSTERTOOLS] can be handy in visualizing and double-checking the network connections between nodes, and seeing which paths VMS prefers between nodes.

[Jan van den Ende]

Richard,

I would like to add a little extra point of attention.
Way back when, when Tom Speake was still the Digital man for Disaster Tolerant Computing, he strongly insisted on the notion that the Cluster Interconnect(s) is essentially your SYSTEM BUS for your COMPUTER SYSTEM (considering the cluster to be ONE system) even if the nodes are dozens of kilometers apart. And he specifically emphasised it in the (then upcoming, now very common) case where the responsibility for "the network" is not in the hands of the VMS system people.... Meaning: insist that you need a DEDICATED network connection for cluster communication. They might get the difference between a network connection and a system bus, even if they look alike.
If I recall our "ip-network" disturbances of the last couple of years, then I am VERY glad we have our dedicated FDDI. We are currently beginning to implement GIG-e. This will probably take the bulk of SCS traffic during normal operation, but we certainly will keep FDDI for fallback. Without it, we certainly would not be 7 years up with a multisite cluster.
So, I agree with Keith: consider investing in FDDI cards. (by the way: are you ADDING your new machines, or doing a rolling REPLACE of your old systems? That is what we did, and we simply re-used our 'old' FDDI cards).

Anyway, SUCCESS!!,
and, if you got it all done, do post a report with your eventual choices, and the story of how difficult (or how smooth) it all went.

Jan

[Tim Hughes_3]

Also check the bufefrs sizes being used. VMS will (I believe) opt for the channel with the largest buffer size before speed considerations. The max buffer size for ALL CHANNELS can be changed via the sysgen param NISCS_MAX_PKTSZ. There is a (proposed??) SCACP switch to control this on a per channel basis but it did not make it into a current releases. So you have to use /Prioity to control which channels are used.

An FDDI with 4Kb packets probably out performs a 100Mb NI with 1.4Kb packets but it would be nice to be able to use both!

Tim

[Guinaudeau]

This thread is old, but maybe Mike's advice could ever be usefull : I confirm : if using 100Mb interface, configure 100/full and NOT autoneg !

Now our experience : we have upgraded to 7.3-2 and upto ECO patch LAN V3.0 a cluster.

ECO patch LAN V3.0 should provide the support for a failover on DE500 devices of DS10. The machine is normally configured to run in cluster (VAXCLUSTER=2) using EWA0
and EWB0 devices [incl TCPIP configured on both with two different IP addresses].

We have experienced the configuration of the LAN failover device for other cluster members
(and on other sites too) without trouble.

After we configured on the DS10 the standard devices EWA0 and EWB0 as a new LAN
failover device, the machine boots but
the machine did not join the existing cluster as expected.

Further, since the failover LLA device had been configured for, it has been used by DECnet and TCPIP, but it sounds like SCS refused to work with !

Finally, after a call by HP, it has been troublesome because the Ethernet device were configured autoneg and not 100/full. That's why i reply in this thread, Mike's advice is worth to look at. The console was serial and nothing helpfull appeared there during the boot sequence, just it "forms" a cluster, not joins the existing one.

Eventually an additional trace level would have been helpfull ?

[comarow]

Don't forget, that while you should have a dedicated network for the cluster, the cluster will continually attempt all paths.

Using mcr scacp you can set the priority of which path the cluster will use. You can even turn off a path if you wish, but then you lose disastor tolerance or temporary problems in your local network.


scacp is a great way to view your cluster activity.

[comarow]

I want to add, unlike tcpip or decnet, cluster traffic is not tolerant of of a degraded network. While tcpip for example will retransmit, if there's a problem on the network clutering will cause clue exits.

It is best for a gigabit private ethernet for cluster scs communication.

[Jan van den Ende]

Like Tom Speake, my Disaster Tolerance teacher said long ago and (by coincidence, in a cosy little long-time-no-see discussion yesterday) assures he still fully endorses it:


"The cluster interconnect is NOT a network component. It is just the _SYSTEMBUS_. A little stretched maybe, but nevertheless.
It should NOT be dealt with by networks people with networks methodologies, but be totally under management and control of the clusters system managers."

For many years now we have lived upon that rule, and on several occasions we have been really glad for it.

Proost.

Have one on me.

jpe