TCP/IP v.5.4 Cluster Load Balancing

[Jefferson Humber]

Want to setup TCP/IP load balancing under TCP/IP v.5.4, utilising the METRIC & LOAD_BROKER services.

It would appear from reading the guide that the LOAD_BROKER writes the metric value back to the DNS servers. Has anybody got this working ?

We use Microsoft W2K DNS Servers, all the examples in the guide mentioned Unix BIND. Have managed to get the BIND Resolver talking to the MS DNS servers, so it would appear that this is compatible at least.

Can anybody out there share their experiences on this matter, does any body have a cluster with balanced IP connections to it ??

Thanks is advance,

Jeff

[Jefferson Humber]

Probably should of mentioned in the previous posting that it's a 2 node cluster we want to balance across using an alias. With the connections coming in from PC's around the company.

[Willem Grooters]

Jeff,

No _real_ experience but to my knowlegde it would be enough to define a cluster alias and connect to that address. Load balance will then be automatic since the node that has least work will respond first - and take the job.

[Jan van den Ende]

Willem:

Yes, simply define a cluster alias...
Those were the days when we used DECnet & LAT: simple, straightforward, and secure.
In todays' IP-dominated world you can forget about that. DECnet is machine-oriented, and you can even have a cluster act as ONE machine. IP is device-oriented, and if you have multople network devices in a system, you even have to specify WHICH of those.
Yes, you can trich a DNS server to periodically change which one to use. This applies for different devices in a machine as well as for different machines in a cluster. If you use round-robin (each address a fixed amount of time, then the next etc), then if one of the addresses in the list is not available, every time this address used, the user or application simply has to wait for a timeout and try again (maybe getting the same address). An improvement is the LOAD BROKER (if your DNS server supports that function). In addition to that, you need a metric "Daemon" (urrrghh) on each node of your alias. Those daemons know the relative power of the machine and the "recent" average load, they communicate with one another to decide who can best carry extra load, en tell DNS to translate the "cluster" address to the network device of that system. In this way that system becomes the cluster "impersonator". Oh yeah... you may strike some extra luck. Your networking people might well cache the name resolution for much longer periods as your impersonator switchovers. Then, one system wil get ALL the load. WE had some real "fun" trying to convince them to use shorter caching times than 24 hours, because "otherwise the switches have to lookup name translations much too frequent, that just costs performance".
If you can manage to have your DNS on your VMS, tou might be a lot better off, but the caching issue remains.

"welcome to the wonderfull world of IP"

Jan

[Lokesh_2]

Hi,

I am not sure about V5.4 , but for earlier versions, we are using cluster alias . See below note to find how load balancing is done when you define cluster alias in DNS:

__________________________________________

Incoming requests are switched among the cluster hosts at the end of each cluster time interval (specified with the SET COMMUNICATION command).


Note

--------------------------------------------------------------------------------
The cluster name is not switched from a host if there are any active TCP connections to the cluster interface on that host.

A remote host can use the cluster alias to address the cluster as a single host or the host name of the cluster member to address a cluster member individually.

All of the TCP/IP services support automatic failover and can be run on multiple nodes in an OpenVMS Cluster. For example, if more than one host in the cluster is running the NFS server, the cluster can appear to the NFS client as a single host. For more information about configuring a specific service for cluster failover, refer to the chapter in this manual that discusses the particular service.

________________________________________

see also help on set commun/cluster_time for information.

Thanks & regards,
Lokesh Jain

[Ian Miller. · Accepted solution]

Don't use cluster address as the node who recives connections to the cluster address only changes when no connections are active (which in practice never occurs).

For the metric server stuff the DNS server holds a name (cluster name) and the IP addresses of each node in the cluster. When asked about the name the DNS server returns the addresses in order of system load - least loaded first. System load is measured by the metric server which the DNS server polls periodically. I've done this with a VMS based DNS server (UCX V4.2 :-) and it was fine because I could control TTL values for the names. You have to ensure the TTL value for the cluster IP name is small enough so nodes do not hold the result of resolving the name too long. Also you need a DNS server that knows how to poll systems for the system load. I know VMS based DNS servers work and I think Digital Unix servers work also. MS DNS servers I know not.

[Willem Grooters]

Jan,

If you re-assign IP addresses to names, yes, you're sure to get trouble with DNS and certainly when expiration times differ. But that has nothing to do with IP or DNS, but more with your configuration. Why would you do that?

For what I've learned in VMS system management and TCPIP configuration:

If a program request data from the _cluster_, it will be served by the least occupied node. The result will carry this sender's address, not the cluster alias. In normal situations quite obvious, even required, since images talk to eachother and each should address the other exactly. So by NODE address, not by CLUSTER address.

Same applies to requests that are issued from a cluster node to a machine outside the cluster (whether this carries a clusetr alias or not). The request will carry the IP address of the sender (that is: the originating node) and the result will be sent to that one. Again: this is task-to-task communication.

Put simple: the cluster alias is for accessing any node within that cluster; the load on each will determine what node will react.

DNS connects name and address. In a clustered environment, you should have 1 name/address pair more than you have nodes: One for each node + the cluster alias one.
Well set up, DNS does it's job AFAIK. I haven't seen or heard of the opposite. And there quite a lot of systems out there that run with a cluster alias. Without trouble.

In case that DNS is to handle load balancing, it's the non-VMS way. I would suggest: DON'T USE IT. Let VMS handle it. Clustering on VMS can handle this, by nature.
(Unless anyone can explain why I should use it. What's the advantage?)

Willem

(FYI: I'm an IP fanatic, I'm well aware of it's shortcomings (don't forget it's old and Unix-ware) but in most circumstances, it's just a requirement)

[Lokesh_2]

Hi Willem,

I agree with Ian. Lets say you have two systems in cluster with ip address x.x.x.1 & x.x.x.2 and on both of them you defined cluster address as x.x.x.3 . Now on DNS , you have defined the cluster name with IP address x.x.x.3 only.

Assume right now there are not active TCP/IP connection to any of the nodes, and I am telneting the cluster with x.x.x.3 . My connection goes to say x.x.x.1 .

Now , as per documentation "The cluster name is not switched from a host if there are any active TCP connections to the cluster interface on that host" , therefore unless I close my telnet connection, all other telnet connection will go to x.x.x.1 only.

So where is the load balancing in this case ? Am I missing something here ?

Thanks & regards,
Lokesh

[Willem Grooters]

ian,

You confuse me.

Don't use cluster address as the node who recives connections to the cluster address only changes when no connections are active

When to be read as:

Don't use cluster address as the node who recives connections to the cluster. Address only changes when no connections are active

In that case I don't see a reason why there would be a cluster alias in the first place. It's of no use if I first have to locate the node in that cluster that hosts the program I'm wanting to connect to.
That cannot be meaning, or am I mistaken?

(Your reply came in when I was typing mine)

Willem

Willem

[Jan van den Ende]

Willem:

we use (have to use) Tru64 DNS, which we cannot manage, we can just request for settings.

>>In a clustered environment, you should have 1 name/address pair more than you have nodes: One for each node + the cluster alias one.<<

Well, nearly correct. If you have redundant network cards, you need an address for each CARD + a cluster alias.
The big fun is, DNS knows your cluster alias name, and of course the corresponding address. In the DNS this is resolved into the MAC address of the current Cluster Impersonator. That's fine, but how does the DNS know? Eighter by assigning it in a round-robin way ( the choice of our Unix people) or by periodically updating by Metric-&-Loadbroker. That requires the right for unattended update, and they are not liking that. As far as I know (please educate me if I miss something, we would be really glad with it) VMS can only apply the cluster alias if it may update the DNS.
And all this still does nothing to prevent the network routers chaching the MAC address to the cluster alias name.
Of course, task-to-task links stay alive for the duration of the communication; as long as there is a link that is used, and not reconfigured.
For outgoing messages (non task-to-task) eg, mail, or any application that uses a messaging communication, nobody has been able yet to tell me how (in DECnet terminology) to "enable outgoing cluster alias". A mail reply should arrive 'somewhere' at any than available cluster node, and not hard linked at the node that happened to be yesterday's sending node that gave its address to the message(which might be not-up for maintenance, or even replacement?)


Jan

[Willem Grooters]

Lokesh,

I don't understaqnd. What is my misconception? By some experience and (mostly) plain deduction:

Suppose: you _know_ telnet is available on the cluster (so x.x.x.3) but you don't know on which node. So you start telnet session to the CLUSTER. Your client (say on x.x.x.40) will request x.x.x.3, but say x.x.x.1 reacts since the service is available there. The connection between your telnet session and the server on the cluster is between x.x.x.40 and x.x.x.1. It will stay that way until you disconnect.
A second user (say: x.x.x.41) will try the same but directly to x.x.x.2. telnet isn't available there so it will fail. If it were, and x.x.x.1 is too busy, x.x.x.2 will respond first, so that connection is between x.x.x.41 and x.x.x.2 - as long as the connection exists.
VERY normal. You wouldn't like your connection to break in the middle of an edit session, would you....
Same applies to FTP. You simply cannot switch nodes during a transfer. As long as your connections lastst, you'll stay on the same machine.

Next: suppose a webserver on both machines. What address should I specify in the browser: x.x.x.1, x.x.x.2 or x.x.x.3? This is just what clustering on VMS is all about: Specify x.x.x.3, and the first server to react will serve you. Again: as long as you connect. For what I know about HTTP, a second request might well go to x.x.x.2 if x.x.x.1 is too busy, since the connection will be set up request after request.

In short: Load balance will occur between connections, not within. Result: An active connection is never broken and will stay on the same node.

[Paul Nunez]

Hi Jeff,

I've not yet used the METRIC server and Load Broker services in v5.4, but I've used them in previous versions.

Some points to remember:

The maxmembers keyword in the load broker configuration file should be _less_ than the number of cluster members participating in the cluster. In fact, I think if you have 3 cluster members, it's better to set maxmembers to 1 (so only the least loaded cluster member's IP address is available.

There will be 'maxmember' entries in the DNS namespace associated with the cluster alias name (you choose). Which IP address(es) are associated with those entries is dynamically determined by the load broker who is then responsible for updating the dns server.

So, at any one time, only 'maxmembers' host entries will be present in the dns name space.

If there are multiple cluster alias entries (maxmembers > 1), the DNS server performs standard round-robin load balancing when queried for the alias name.

The least loaded cluster hosts (as measured by the metric server) will (typically) be the ones available in the DNS name space.

So periodically the load broker receives (or requests) the load metric from each "cluster" member and (iirc) then it queries the dns server for the list of those cluster members the DNS server currently has associated with the alias name and makes any necessary updates.

"Clients" often cache dns responses, so setting an appropriate ttl on the alias name is sometimes a necessary endeavor(ymmv).

HTH,

Paul Nunez
HP Services

[Willem Grooters]

Starts to look like a crash course on the subject :-D

Jan:

>Well, nearly correct. If you have redundant network cards, you need an address for each CARD + a cluster alias.

Ok, I missed that, but it doen't change my view. One, two, three ... names and addresses, one or more NICS - All the same.

> For outgoing messages (non task-to-task) eg, mail, or any application that uses a messaging communication, nobody has been able yet to tell me how (in DECnet terminology) to "enable outgoing cluster alias

FAIR. This might be a problem and I know of situations where it is.
Mail is however a bad example, since this connects once BY NAME (MX, DNS, Host) and will exchange all messages in one connection. Matter of the right sepcification of the mail server: the cluster alias!

Willem

[Ian Miller.]

re IP cluster alias - I have found it of limited use. Its ok for services such as use ftp to send a file to the cluster but does not seem to work for interactive users (to balance the load). Are the metric and load_broker services both required or are they two ways to do load balancing of incoming connections?

[Jan van den Ende]

Ha die Willem,
het gaat nog gezellig worden!
and I ain't got no bigger smiley than you, makes me sad :<[

>>>
Mail is however a bad example, since this connects once BY NAME (MX, DNS, Host) and will exchange all messages in one connection.
<<<

I'd like to see that! Are you implying, when I send you mail, that that connection stays open until you might or might not decide to reply, maybe now, maybe tomorrow, maybe when you return from your holiday?

No, the mail goes out, specifying, like you said, by name, BUT, that name is NODE_x, and NOT MY_CLUSTER. And with some luck, our cluster has been extended last week by the latest Marvel NODE_z, which becomes fully operational later this afternoon, after what we shut down 6-year-old NODE_X "forever" to be configured out.
IF mail used a cluster alias, any non-system-management user would not even notice, but in the IP world...
THAT is what I meant by ENABLE OUTGOING ALIAS.

[Jefferson Humber]

Wow, that's a lot of responses.... must really be onto something here.

As already discussed the 'IP Cluster Alias' within TCP/IP acts with one node becoming the impersonator, only passing the priviledge to another node when no active connections are present. As quite rightly mentioned this is useless for interactive users, all your end up with in a heavily loaded VMS box..... with it's cluster partners empty.

It seems under v.5.4 you can run a 'Load Broker' on each box in the cluster, a big improvement over previous releases where you couldn't.

From reading all the replies it would seem that nobody is doing this with Microsoft DNS servers.... I feel like a guinea pig on this one, will let you know how I get on.

From looking at the OpenVMS roadmap further cluster support is being added with each new release, but the dates mentioned are Q2 2005 & Q2 2006.... which is a long time to wait.

Thanks very much to Ian & Paul, Paul I would be interested to here more about how you configured your example under v.4.2

Anybody got any good TTL values to reccomend ?

Cheers,

Jeff

[Mike Naime]

We use the cluster alias for "failover" purposes.

We normally run 2-node production clusters where the main Application/Oracle processes run on node one, and some app/oracle and any/all interface proccesses run on node 2.

By use of the cluster alias, we can easily switch everything to run on node 1 when the second node is no longer up. We do not have to re-program the far end of the interface systems to point to a new IP address.
If the first node fails, you start all the application processes on node2.


Mike Naime

[Jan van den Ende]

Mike,

to me that is not how I have become accustomed to clusters: a cluster to me means everything is available everywhere in the cluster: the "share-everything" concept.
I know to most non-VMS clusters this sounds like impossible nonsence, but we are using it for nearly 20 now. A fail-over construction like you describe is what most Unixes and even Billy-stuff are approaching, but it wil NEVER be able to share the load.
However, if it serves your need, that's OK isn't it?
In an homogenous VMS cluster you don't even CARE what runs where, and you don't bother with failover (unless you run databases that have been ported of course.. :-) ).
Native VMS databases like DBMS & RDB access the databases simultaneously, directly from very node.
Actually, Oracle is the exception here, that can run on multiple nodes simultaneously (is this still correct? my latest hands-on Oracle experiences are ORA-V6, when it was still VMS-develloped stuff. Did they keep it in now that it is ported from Unix?).
So Mike, in your case I guess the IP cluster alias suffices. For my curiosity: How do you get the Impersonator to fail over? Looks to me you will have to kill ALL IP connections to the current Impersonator, even those that have nothing to do with your ORA-app?

Willem:
my last remark about mail being sent by a node, and NOT the cluster? It's even worse:
we got multiple network cards, and the mail sender address is A SPECIFIC CARD on A SPECIFIC NODE!

Jan

[Willem Grooters]

I have the impression that the main problem seems to be the presencse of an impersonator of the cluster.

I think Jan is right. In the true VMS sense, you wouldn't need one. ANY node in the cluster could serve a request, and the one least occupied would be the first, not limited to protocol. Truely "share everyting", and real auto-load-balancing.
I know that in that sense, you can login to the cluster but it is uncertian on which node you will actually be logged in.

Outgoing traffic would, only when required, send out the node's address (task-to-task connections), otherwise the cluster's address (I don't know whether DECNet would have that behaviour, but I imagine this would be the case).


This 'democratic' - some say: anarchistic - behaviour of VMS clusters is unknown in the Unix world, which is the base of IP. They talk about "master" and "slave" (aka "client"-"server"). Hence you need to have one system to talk to: the impersonator. For a cluster, it would mean you need to point out some node to be impersonator at some point.
If you login, you can NOT login at the cluster but to a given node. It's absolutely clear which node you login to.

If this is true, there should be a way to be able to keep current connections where they are, but new connections to be located to the other node.
But most of all: I think that engineering has made a really BIG mistake by introducing the Unix way of clustering - in stead of sticking to the way VMS is doing it. For us, the only (right) way.

Willem

[Ian Miller.]

Re use of IP cluster alias for failover. I've done this on earlier versions and its fine for certain limited purposes - its not load sharing. What happens is node A owns the cluster IP address and responds to connections and after X seconds, if no connections are active then node B owns the cluster IP address, and back and forth. If there are active connections it stays with the node. If a node crashes the other node will own the cluster IP address and respond to connections. I used to use it for sending files to a cluster using ftp. The sending node would connect using ftp to the IP cluster alias and send the files. It did not matter which node and as long as a node was running the ftp succeeded.

For a description of how load balancing works in current TCPIP V5.4 see section 3.2.2. in
http://h71000.www7.hp.com/doc/73FINAL/6523/6523pro.html

On UCX V4.2 the DNS server returned a list of IP addresses when asked about the cluster name. The IP address list was ordered by least loaded first. the DNS server sent queries to the metric server on each node defined to be in the cluster. How it is set up on a UCX V4.2 DNS Server is that you define the cluster name as a A record with multiple IP address then do UCX SET CONFIG BIND/CLUSTER=clustername . The metric server had to be enabled on each node whose IP address was listed. They did not have to be in a VMS Cluster.

[Jan van den Ende]

Hey Ian,

well, basically what happens with us looks a bit like it, only even less sofisticated.
Our DNS is on TRU64, maintained by them, and
what they will offer us is a round-robin of time-slices, alternating the address given to our clustername over the various network-cards of the various cluster nodes.
So, we do have a load-spread, but only logintime based, and certainly not load-based. Actually I am not sure if the our clusteralias is used at all.

Willem:
you DID make me thinking.
would it really be SO much trouble for engeneering to activate OUTGOING ALIAS in IP?
Maybe the IP technicians would have to consult their DECnet colleagues, but that should not impose impossible hurdles.
And DON'T let them get out the shit about violating standards: not any more than VMS filenames, VMS file structure (if communicating with other VMS) etc. It is just implementing another VMS-extra feature.

If they really can follow up on this, then REALLY something good his come out of this!

[Jefferson Humber]

Have managed to get the Load Broker working with our Microsoft DNS servers.

I am now load balancing IP connections across the cluster with a metric.

The configuration I am running with at the moment is as follows;

cluster "mycluster.com"
{
dns-ttl 45;
dns-refresh 31;
masters
{
;
};
polling-interval 10;
max-members 1;
members
{
;
;
};
failover ;
};


Have successfully tested the alias address switch from one node to another, by artifically loading them.

Many thanks to you all for all your advice and pointers,

Jeff