Replacing Memory Channel with Gigabit Ethernet

[Andrej Jerina]

We have OVMS 7.3-2 cluster with 2 ES40, 1 ES45 and DS10, conecetd with Memory Channel and Gigabit.
We will add ES47 and replace Memory Channel with another Gigbit NICs on all nodes.
Also we will move 2 ES40 to remote location few hundred meters away.
We are afraid, that replacing MC with Giga will reduce performance and slow down production.
Any advice on what we have to care about (like changing any parameters) is appreciated.
Is any sense, to remain MC only between main production systems (ES47,ES45,DS10) ?

[Ian Miller.]

If you introduce Gigabit Ethernet between all nodes then you should consider the use of jumbo frames especilly if you use host based shadowing. The cpu usage of the ethernet driver is higher than the memory channel driver and the latency is higher.

Keeping the MC between some nodes may help depending on the way the workload is distributed.

[Ian Miller.]

A measurement of VMS distributed lock request latency was done and the result showed 120 MicroSeconds for Memory Channel and 200 Microseconds for Gigabit ethernet.

This means remote locking is slower with GBE compared with MC. The impact of this on your cluster depends on the way locks are used.

See
http://h71000.www7.hp.com/freeware/freeware60/kp_clustertools/
and
http://h71000.www7.hp.com/freeware/freeware60/kp_locktools/

for various tools to monitor your cluster.

[Thomas Ritter]

Gigabit Ethernet is the way to go. We run a split cluster of 4 ES45. The two sites have two fibre cables of 8 and 4 km in length. Locks peak at about 2,500,000 and remastering issues are not a problem. Good workload management will prevent constant dynamic remastering.

IMO move away from propriety solutions like Memory Channel. View the interconnects as networks and take advantage of all the good networking devices out there.

Tom

[Peter Quodling]

Memory Channel, was designed more around latency issues than Bandwidth issues. Do not confuse the two...

q

[John Gillings · Accepted solution]

Andrej,

With cluster interconnects, it's usually a case of more is better.

If you don't have a compelling reason for disconnecting the memory channel, then leave it alone, at least for the nodes within the distance limits.

If there are more unused network adapters, then just connect them all - for example, a "private" hub with some or all nodes connected. No need to configure them, cluster software will automatically find the paths and make use of them. Similarly if you have unused 100Mb NICs, switches and cables are very cheap, and they provide redundant connections between nodes.

At the very least, the fastest path will be used. With more recent versions of OpenVMS will load balance across all available interconnects.

For future planning, I agree with Thomas. Go with Gb ethernet, rather than MC as a cluster interconnect.

[Robert Atkinson]

We did this a number of months ago, after years of problems with MC.

I'm glad to say everything went without any problems.

As John said, use a private network for the Cluster traffic. We used 2 CISCO switches, with links to each node in our 3-node cluster.

Setting DECNET and Cluster traffic to use just these paths was a bit fidly, but worth it in the end.

2 of the nodes are 75% loaded ES40's, and I've not seen any lock manager issues so far.

Rob.

[Ian Miller.]

keep the GBE used for cluster traffic away from the network people - they often don't understand the availability requirements or the fact that its not IP and can cause trouble.

[comarow]

An interesting factor. While 100 mb cards were recommended to hard code fast full duplex, as digital/hcompaq didn't conform to standards, on gigabit ethernet use autonegotiate.

If you use a Cicsco swith you can do a show tech and it will show the settings on all the ports without any loss of security.

Bob

[Colin Butcher]

I'd suggest dual Gigabit ethernet with two physically separate private GigE LANs (not VLAN's out of a big switch, use one dedicated small switch at each site) for the cluster traffic, then use other LAN interfaces for 'user' traffic (TCPIP, DECnet, LAT etc.).

How you set it up will depend very much on your workload (eg: locking implications) and other system features you're using (eg: HBVS). Look at the storage subsystem, how that's connected and what kind of load the application imposes on the storage subsystem. Is it a disc IO intensive application, or CPU intensive, or LAN IO intensive, or whatever.

In general dual GigE seems to work pretty well compared with MC. As Ian mentioned - jumbo frames can help - which is another good reason for making the interconnects private and for clustering use only.