topic Re: $WAITFR behaviour in Operating System - OpenVMS

$WAITFR behaviour

Barry Alford — Mon, 29 Oct 2007 06:46:09 GMT

I am trying to synchronize two processes using event flags. One process (the Master) runs a wait loop using one event flag in a timer ($SETIMR). Once the timer event flag is set, it is cleared and another event flag is set at the end of the loop. This second flag is then cleared at the top of the loop and the timer is reset:

repeat
$SETTMR(timerFlag, cycleTime)
$CLREF(eventFlag)
$WAITFR(timerFlag)
$CLREF(timerFlag)
$SETEF(eventFlag)

This is intended to form an "escape mechanism" - the two flags are never both true.

A slave process implements this loop:

repeat
$WAITFR(timerFlag)
$WAITFR(eventFlag)

Which is intended to keep time with the master process. This relies on $WAITFR having this (documented) behaviour:
"$WAITFR - Tests a specific event flag and returns immediately if the flag is set; otherwise, the process is placed in a wait state until the event flag is set."

I took that to mean that when the flag is set, the process will definitely be woken up and run and see the flag as set. However, I do not see this happening; it seems that because the flags are only set for a short period that these events are "lost" and the slave processes run very erratically (>1 in 10 master cycles) as if the flags are being polled by the processes and not being triggered by an event from the OS.

Have I missed something here?

Re: $WAITFR behaviour

Richard Whalen — Mon, 29 Oct 2007 08:27:20 GMT

Assuming that timerFlag and eventFlag are in a common efn cluster, then I would expect this to work. If you find that it is erratic, then I suspect there is a bug in the version of VMS that you are using.

What version of VMS are you using?
Are all the patches installed?
Is this a multiprocessor system?

Re: $WAITFR behaviour

Hein van den Heuvel — Mon, 29 Oct 2007 08:36:13 GMT

My we assume these are some sort of common flags?
Is this a new design? Testing on a multi-cpu system?

>. Have I missed something here?

Yes, you missed a glaring timing window.
Once the timer flag is set, the slave is officialy runnable.
The waitfr time is done, and the waitfr other is about to be requested.
But in the mean time the master sets the other, arms the timer, clears the other and waits for the timer.
The scheduler now starts workgin on the slave.
It finally actually executes the waitfr other, which at this point is cleared. So it really waits for the timer.
Once cycle missed!

KISS!

One event flag is genrally too hard to deal with already.
Be sure to check out $HIBER / $WAKE instead.
Unlike event flags the pending wakes are rememberd.
Much easier!

Good luck,
Hein.

Re: $WAITFR behaviour

Barry Alford — Mon, 29 Oct 2007 09:22:32 GMT

I have tried it on v7.2-1 and v7.3-2 (both on DS10 single processor) so far and about to try v8.1 (PersonalAlpha). All will be pretty much unpatched :-o. (I am reluctant to believe that something as fundamental as this would be broken!)

I am using a common event cluster:
[in Fortran]
$ASCEFC(%VAL(iEflag), %DESCR("TIMER"), %VAL(0), %VAL(0))
...for both flags (69 & 70 in fact)

Hein, I see your point but I think that would make the slave miss at most one firing of the eventFlag. Consider the flags are doors into and out of a room - once the slave enters the room (on the timerFlag) it's waiting for the exit to open (on the eventFlag). Meanwhile, the master has opened the exit many times but the slave doesn't come out!

The problem of using $HIBER/$WAKE is that slaves will have to register with the master to get woken up; I wanted to keep things more adhoc...

(The processes will, in fact, map to a shared region of memory, but I wanted to find a general algorithm. Back to my old college text books and refresh my hazy memories of p & v and co-routines?)

Re: $WAITFR behaviour

Hein van den Heuvel — Mon, 29 Oct 2007 09:36:41 GMT

Yabut... the one missed 'other wait' is but the beginning. The slave will also miss a timer event, while waiting for the other flag. So that's 2!

Be happy that you tested this on a single CPU system. You might not have found the design problem on a multi-cpu system untill way too late, but it would have been equally broken!

>> (I am reluctant to believe that something as fundamental as this would be broken!)

Ah, give yourself 8 points!
That would have been 10 points if you had written 'refuse to believe'.
[Yeah, I know you can not give points to yourself]

When reading the base topic, I half expected to read 'waitfr' is broken, and was pleased to see that was not mentioned but replaced by an 'have I missed something'. Excellent.
Now I see it was fully intentional and it pleases me.
There are too many daft individuals out there that think their first dables must have uncovered a major flaw in fundamental stuff. Not!

Cheers,
Hein.

Re: $WAITFR behaviour

Barry Alford — Mon, 29 Oct 2007 10:54:14 GMT

Ah Hein! I'm not awarding any points just yet!

Well, well! When I monitor the slave process with:
$ SHOW/PROC/CONT/ID=
.. it all works _perfectly_! Stop monitoring, and it all goes sticky again.

How d'you like them apples? :-)

Re: $WAITFR behaviour

Hoff — Mon, 29 Oct 2007 11:34:27 GMT

Have you missed something? Yes, you've missed that event flags are an OpenVMS analog of "die Lorelei", or of Homer's Sirens. A construct that serves to lure unsuspecting programmers onto the rocks of pain and suffering. By sheer coincidence, I posted up a similar statement to this one -- and a description of why you're headed for the rocks -- just last night.

http://64.223.189.234/node/613

Event flags only look simple. They can get to be very nasty, this in terms of spurious triggers, problems with scaling, limits on the numbers of parallel events, and otherwise.

On no details on the application, I might tend to use locks and potentially lock value blocks here. Mayhap shared memory. I'd work to keep time with the master, whatever that means here -- and some details and some background on the application synchronization requirements would be useful.

Stephen Hoffman
HoffmanLabs LLC

Re: $WAITFR behaviour

Barry Alford — Mon, 29 Oct 2007 12:05:48 GMT

Thank, Hoff, for the warning. I will look up that link tonight (restricted access here).

All I asked was for clarification of how $WAITFR worked; it seems from your words and me scraping my ship on the rocks that the Land of Event Flags is not the place for me!

The application does simulation of various machines; currently each machine is processed serially in a time step. This makes changes a problem in that the whole app has to be rebuilt. We have toyed with shared libraries and late binding, and my aim with this exercise is to experiment with multiple processes, each simulating one machine, but running in step with each other in time. (Did someone say "threads" ot there?)

Anyhow, I will now try Plan B: use the master timer to wake up processes, then a cycle number in shared memory to ensure only one processing step per master cycle.

Re: $WAITFR behaviour

John Gillings — Mon, 29 Oct 2007 16:22:33 GMT

Barry,

I agree with Hein and Hoff. Event flags are are very hard to get right. They tend to have nasty timing windows, and because there's so few of them they often get overloaded, so you have to deal with spurious wakeups.

Consider using a pair of locks, maybe called TICK and TOCK. You can lock step your processes by cycling the locks converting to EX then NL in sequence. Put your cycle number in the lock value block.

Now, since your locks are exclusive to the specific pair of processes you avoid any logic for spurious "wakes", and you're guaranteed handshaking. Moreover, since they're locks, the mechanism will work across a cluster (and it can be scaled up to multiple processes fairly easily - just add another TICK for each slave). With some extra logic on the lock value block, you could also build in a way to monitor the presence of the other process.

Master
repeat
$ENQ TOCK CVT->EX ; wait for slave
$ENQ TICK CVT->EX ; block slave
$SETIMR
wait for timer
$ENQ TICK CVT->NL ; release timer
prepare for slave to be released
$ENQ TOCK CVT->NL ; release slave
slave is now executing
next

Slave
repeat
$ENQ TICK CVT->EX ; wait for timer
timer complete
$ENQ TOCK CVT->EX ; wait for master
do something
$ENQ TOCK CVT->NL ; signal complete
$ENQ TICK CVT->NL ;
next

Re: $WAITFR behaviour

Robert Gezelter — Mon, 29 Oct 2007 17:47:17 GMT

Barry,

I would agree with John, except it is not clear to me that you actually need two locks to accomplish this.

Personally, I would do this with locking and ASTS to make things maximally safe.

Doing this with event flags is tricky, as has been commented on.

- Bob Gezelter, http://www.rlgsc.com

Re: $WAITFR behaviour

Hoff — Mon, 29 Oct 2007 18:19:48 GMT

To add one other oft-overlooked interprocess communications mechanism that's available on OpenVMS: RMS. RMS can be a silly-fast communications channel for many applications, and with very minimal configuration requirements.

Re: $WAITFR behaviour

Claus Olesen — Mon, 29 Oct 2007 21:08:02 GMT

To me it looks like a problem of who gets the cpu. OpenVMS will put the slaves on the runable queue when the master sets the ef but the slaves may not actually get the cpu for while after that. All while the master is free to continue in its endlessly loop of setting and clearing the efs whether or not the slaves have seen it. You may not be able to use realtime priorities but if the master could be set at a lower realtmie priority than the slaves then that could ensure that the slaves gets to run all the way up to the next eventflag before the master again gets the cpu.

Re: $WAITFR behaviour

Hein van den Heuvel — Mon, 29 Oct 2007 21:34:56 GMT

Absolutely Claus.

That's a much better and simpler way to explain it. Occam's razor.
As presented the master never waits for the slave.
There is no formal synchronization at all.
It's just 'likely' that the slave gets scheduled when the master goes to wait for the timer, but it is not garantueed.

The $show proc/cont just changed the priorities and scheduling.

Cheers,
Hein.

Re: $WAITFR behaviour

John Gillings — Mon, 29 Oct 2007 22:22:40 GMT

Re: Robert "except it is not clear to me that you actually need two locks to accomplish this."

If there are only two processes, you're absolutely correct. One lock will suffice, but it seems to me there may be several "Slaves" here. For maximum control, and to guarantee each slave gets a look in, have one lock for the master, and one for each slave. The slaves synch against master and their own lock. The master controls them all.

re: Hoff's comment about RMS - absolutely true. RMS files can be a simple way of using locks from DCL.

Re: $WAITFR behaviour

Barry Alford — Tue, 30 Oct 2007 03:56:51 GMT

Thanks to all for your replies -- I didn't think that this would elicit such a discussion; I thought I had just misinterpreted how event flags, and especially $WAITFR, works.

Claus solved the problem by mentioning priorities; if I run the master at base priority 3 and the slaves at the default of 4 (tried it with two slaves so far). Hein suggested that the $show proc/cont changed the priorities, and this looks like it may be the case - but (like in quantum physics) I can't measure this without changing it!

I have not tried locks/ASTs as suggested by John and Robert; I may explore that later.

By the way, my Windows programming collegues tell me this is really simple in their world! However, I believe their event driven code is like the slaves registering with the master which Hein suggested with his $HIBER/$WAKE, albeit with some more help from the OS.

Re: $WAITFR behaviour

Hein van den Heuvel — Tue, 30 Oct 2007 06:11:22 GMT

>> Claus solved the problem by mentioning priorities; if I run the master at base priority 3 and the slaves at the default of 4 (tried it with two slaves so far). H

NO NO NO NO NO

Assining priorities does not solve problems it merely ides problems.

IF strict sncrhonization is a requirement, then the master somehow has to listen to the slave.

IF it's ok to miss a cycle or two then priorities can be good enough, but they are never a solution.
For example, at some point the maste may do an IO an get a priority boost from that. Or pixscan kicks in or whatever.

Since you have a time goin of anyway this could be acceptable in your case, as long as slaves look for all possible work, not just 1 item.
The folks with just one event event flag and no time may hang too long waiting for the action if a wakeup is missed.

Cheers,
Hein.

Re: $WAITFR behaviour

Barry Alford — Tue, 30 Oct 2007 06:54:59 GMT

Hein, maybe I should have said that Claus had solved the problem of the slaves not seeing the setting of the event flag -- I don't mean to say that this is a solution for my application.

It /could/ be a solution, as some missed frames may not be critical - the slaves are intended to run at 50Hz (20ms cycle time) as their position info is monitored by a graphics process (running on a pc). (Currently, the monolithic application does all that the slaves do in this period.) I have not yet run at this rate or given the slaves any work to do; this may then show up the flakiness of the scheduling problem again...

If so, I would probably try the $HIBER/$WAKE solution suggested by Hein; then I have control of the "wakeup list" of process IDs, rather than depending on what I thought $WAITFR would do. (I will need a mechanism for the slaves to register their process ID with the master.)

Finally, I may also try setting the slaves free and run them at 50Hz but asynchronously to each other. This may be no worse than tolerating them missing a cyle when running under the master.

Re: $WAITFR behaviour

Hoff — Tue, 30 Oct 2007 12:07:38 GMT

Why are the secondaries even having this problem?

(There's an ancient software design rule I ascribe to: if it hurts, don't do it.)

Consider another approach... Have a scheduled AST or $schdwk or other such in each of the secondaries (repeating TQE), and free-run the secondaries pulling in the frames from whatever the shared storage is.

The other approach I've used in this environment is an Ethernet multicast. I have used 60-06 here when I first started with this scheme an eon ago (as that stays on-LAN), but I'd also look to use an UDP multicast datagram for various environments.

http://h71000.www7.hp.com/DOC/82final/6529/6529pro_005.html

Basically, the primary multicasts the data (or just a ping) periodically, and if the secondaries pick up on the multicast, they display the data. So long as the next datagram is captured and the data in the datagrams are independent, if a UDP multicast datagram is occasionally dropped on the floor, well, oh, well...

I've successfully used this or a similar scheme for any number of monitoring tasks over the years. (It's also entirely platform-independent; it works on OpenVMS and on anything else that can receive a UDP multicast.) (If you want to chat off-line, I can describe a boo-boo or two I've made when designing these sorts of factory-floor and other such comm systems over the years.)

As for coordination and the election of a primary process, that should use locks. Coordinating processes within a cluster through other means is perilous at best, and a whole lot more work to get right. I'd not use $hiber and $wake for this task, and would definitely not use event flags here.

Re: $WAITFR behaviour

John Gillings — Tue, 30 Oct 2007 16:21:35 GMT

Barry,

Fairly simple rule... any synchronization mechanism that depends on differential priorities is wrong! It WILL break somewhere, sometime, when you least expect it, and you won't be able to reproduce the failure.

>By the way, my Windows programming
>collegues tell me this is really simple in
>their world!

Synchronization between processes in OpenVMS is also really simple. Any operating system is fundamentally just a big synchronization engine, so it can't be any other way.

I suppose it could be argued that on OpenVMS it's made slightly harder because you have so much choice as to what mechanism you pick. So, rather than having to bend your design to fit what the operating system offers, you choose the best design for your application and then select the most appropriate mechanism to implement it.

Event flags can be good for "blind" signaling. "Wait here until something happens". There can be multiple processes all waiting on a common event flag, and they all get released at once, BUT you get no feedback, and no way to know all processes are ready before lowering the gate. They can be a good, light weight mechanism, but unless you're very careful you may introduce timeing windows, race conditions or false triggers.

Other mechanisms include HIBER/WAKE, locks, mailboxes, global sections, ICC, IPL, spinlocks, mutexes, ASTs, etc... See the Programming Concepts Manual.

Your first step is to make sure you can clearly describe what you want to do. I'm not sure I know that yet. I THINK it's like this:

master process
timed loop
send an event to each slave
confirm slave has received event
endloop

slave (multiple processes?)
loop
wait for event
confirm receipt of event
endloop

There are lots of possible ways to do this, but event flags probably aren't the best choice.

Re: $WAITFR behaviour

Claus Olesen — Tue, 30 Oct 2007 20:59:33 GMT

Hoff suggested using locks. I tried it using this

lksb A,B,C;

if (!strcmp(argv[1],"master"))
mode=LCK$K_EXMODE;
else //slaves
mode=LCK$K_CRMODE;

for (x=0;;x++)
{
printf("%d\n",x);
sys_enq(A,mode);
sys_enq(C,LCK$K_NLMODE);
sys_enq(B,mode);
sys_enq(A,LCK$K_NLMODE);
sys_enq(C,mode);
sys_enq(B,LCK$K_NLMODE);
sleep(atoi(argv[2])); //your work here
}

(sys_enq is pseudo for your convenience wrapper of sys$enq) and my test run with one master and 2 slaves showed lock step without missteps. It has that characteristic that you mentioned that the parties do not need to know about one another. And they can come and go.