Real-time process control in an OpenVMS Cluster environment

[Keith Parris]

There is a perception that OpenVMS Clusters can't be used for real-time process control. I think this deserves closer examination.

In a recent thread entitled "VMS Failover" was this post by my good friend Stanley Quayle:
> I've been involved with doing redundant VMS systems that aren't clustered -- to avoid the cluster transition time. The time can be made smaller, but not zero. There are some applications that can't take that. Process automation, for one. <

I did a DECUS presentation years ago which discussed cluster state transitions and how to minimize their impact.

Failure detection can be set to as low as one second, by lowering TIMVCFAIL to 100.

Ride-through of a connectivity problem can be set to as low as 1 second by lowering RECNXINTERVAL.

So if your realtime process HAD to use distributed locking, you'd be forced to wait for a lock for a maximum of a second or two.

If it is the potential for quorum loss in a clustered environment that you're concerned about, one option to consider is to set up your realtime process so that it itself doesn't do any disk I/O, locking, or anything else that depends on cluster coordination (in other words, it would talk with the devices it controls but perhaps communicate through memory with another process which IS under coordination by the cluster to do disk I/Os if needed), and then you could remove the requirement on that particular process for the QUORUM capability bit, so that it could continue to run uninterrupted through a period of quorum loss.

[Robert Gezelter]

Keith,

It all depends upon the context. I do not have time to write a long commentary at this instant, however I will make one or two points and try to get something more extensive either here or in my column.

Many process control environments are VERY timing sensitive, particularly control systems for industrial processes. In these cases, one second is a very LONG time.

In my practice, I have encountered the entire spectrum, from applications which are perfectly suited to OpenVMS clusters, to applications which are so realtime that even the system timers are too imprecise. They key is careful analysis of the requirements, and what the different impacts are.

That said, I have seen a sub-optimal choice made, either for or against OpenVMS clusters. The most important issue is the proper analysis.

Hopefully, I will have the time to write more later.

- Bob Gezelter, http://www.rlgsc.com

[Stanley F Quayle]

> [...] and then you could remove the requirement on that particular process for the QUORUM capability bit, so that it could continue to run uninterrupted through a period of quorum loss.

And how do you do that?

[Keith Parris]

> a DECUS presentation years ago which discussed cluster state transitions

Sorry -- I meant to provide a pointer: see http://www.geocities.com/keithparris/ and look for "Understanding VAXcluster State Transitions" and "VMS Cluster State Transitions in Action".

> > remove the requirement on that particular process for the QUORUM capability bit
>
> And how do you do that?

The routine SCH$RELEASE_CAPABILITY (found in module [SYS]CAPABILITY.MAR in the sources) is what is used to do this by a process running OPCCRASH and by the lock manager server process (used for dedicated-CPU lock manager operation). Of course, we're talking high-IPL code, holding the SCHED spinlock, etc. -- not for the faint of heart. But it does look possible.

[Dan Cornett · Accepted solution]

As noted earlier, it depends on the requirements of the usage.

We have used VAX clusters for "level 2" process control -- what is sometimes termed supervisory control -- for many years. The key has been that the crucial tasks for the "real-time" portion do several things:

a) They don't do any disk I/O themselves ... they pass (or recieve) data via memory resident commons (implemented via a locally writen API that acts like "mailbox queues" - a message passing subsystem) to other tasks which buffer the I/O to disks ... or to displays (which might get XOFFed)!

b) Those tasks run somewhere above priority 16 (and one keeps the amount of "work" done very short ... only the crucial real-time portion, of course).

That said, these uses have been very successful (and virtually entirely in Fortran!)