OpenVMS 8.3-1H1 Itanium SYS$SCHDWK call

[Dan R Farrell]

We are using a SYS$SCHDWK call to run a process 50 times a second. It works except every six hours from boot time it misses a few cycles. This is very repeatable. It does not matter if the process is running for hours or for a few minutes. Six hours after boot time and every six hours after that (almost to the millisecond) it misses cycles. I think the system time may be getting reset but we have shut down almost every on the system except for VMS and network and it still happens. NTP is not running. Has anyone seen this issue before?

[Robert Gezelter]

Dan,

Have you verified that the computation of wake times does not have some slip in it?

- Bob Gezelter, http://www.rlgsc.com

[Richard Whalen]

50 times a second is very often!

I know that VMS on ia64 (not necessarily 8.3-1H1) stores the value of the TOY clock on disk approximately every 6 hours. I suspect that your problem is related to this.

[Jon Pinkley]

Dan,

Anything that can block scheduling has the potential to cause a wake up to be delayed.

Or as Bob suggested, depending on how you are computing the next wakeup, you may be getting rounding errors. For example, the LIB$CVTF_TO_INTERNAL_TIME may be subject to rounding errors with small intervals. If you are using integer math, then I doubt that is the cause of the problem.

There are several possibilities, either higher priority processes could be preventing the process from being scheduled or some high IPL code could be blocking process scheduling or even the hardware clock interrupt.

I am not sure how often the BBW battery backed up watch gets updated, but hopefully that couldn't cause cycles to be lost, and I wouldn't expect any disk I/O to be blocking scheduling.

If it is that repeatable, I would fire up the PRF SDA extension to collect samples starting 10 seconds or so prior to a 6-hour epoch and see what is happening. I'm reasonably sure it isn't driven of the HDWCLK interrupt so I believe it has a chance at seeing code executing at HWCLK IPL. If PRF is using EXE$GQ_SYSTIME in its time stamp calculations, it may lead to false conclusions about "when" something happened.

Do you have other timer based code running, or some performance data collector that could be running something at high IPL periodically?

Jon

[John Gillings]

Dan,
Just to clarify...

I'm assuming you're calling $SCHDWK with a "reptim" value of 20msec, as opposed to calling it every cycle with a "daytim" of 20msec?

Could you show us the actual code?

A few things I'd worry about...

1) first 20msec is only 2 quanta. I'd want my value to be as close to the real value as possible. If I cared about it a lot, I'm not sure I'd trust a $BINTIM conversion to do that for me. I'd be checking the bits in the time value.

2) The timing of the $WAKE has no influence on when the target process responds (ie: actually wakes up). How can you distinguish between the $WAKE being late, and the process "sleeping in"?

3) $HIBER/$WAKE seems like a rather blunt instrument to use if you require high precision ticks. Maybe you should consider other possibilities? For really accurate, high frequency timing, you pretty much have to dedicate a CPU and busy wait.

Things to try...

What happens if you double the frequency to 10msec?

If you haven't done so already, build an absolutely minimal test program. On waking, don't do anything other than sample the time and put the results in a ring buffer.

Are you running with multiple CPUs? Have you tried using affinity?

[Hoff]

I'm actually somewhat surprised this works as well as this does and you're only seeing a few cycles every six hours; this looks to be a polling-based design, though somewhat cloaked in the garb of a multiprocessing application. And I'd expect to see a few cycles going to other tasks here and there.

I might well look to abscond with a core here and go to full-on polling, rather than a 50 Hz (60 Hz in the US?) solution. That, or (depending on what is going on) I'd look to start dealing with the cruft in an out-board processor here, as those are cheap. There are also ways to release the processor through the scheduler interface, too.

Do call HP, as they're the arbiters of this sort of thing and (if you're doing 50 process activations a second) you probably have a support contract.

[Jon Pinkley]

I hope by "run a process" Dan didn't mean an image activation. I assumed his process was scheduling a wakeup and hibernating. For that, 50 times a second shouldn't be taxing things (on average), as long as his process (kernel thread) software priority is in realtime range. I don't think VMS claims to be REALTIME, at least in the general case, and if this node is part of a cluster, then all bets are off.

Dan, if you really need something hard scheduled 50 times a second, I would be looking at a dedicated collection box that can weather the peak demands, cluster transitions etc.

John, I wasn't aware that the VMS schedular waited until quatum end to reschedule a sufficiently higher priority process. If it does, then either things have changed, or my memory is incorrect.

Jon

[John Gillings]

re: Jon, "or my memory is incorrect."

Sorry, maybe I wasn't clear enough. My remark "20msec is only 2 quanta" wasn't referring to the SYSTEM parameter QUANTUM. I was referring to the limit of the "reptim" parameter:

(from docs) "The time interval specified cannot be less than 10 milliseconds; if it is, $SCHDWK automatically increases it to 10 milliseconds."

The issue is potentially one of granularity. When you're down at that level, even small absolute errors in calculating time intervals can be large percentage errors.

It's also unclear from the documentation if 10 msec is just a lower limit, or a granularity. Would a request for (say) 14msec be rounded up to 20msc or down to 10msec?

When you're this close to the documented limits, and you care enough about the exact behaviour to ask a question like this one, I'd be strongly recommending having a look at the sources to see exactly how $SCHDWK uses its parameters and calculates the time intervals to generate the $WAKEs.

Always remember, a computer is NOT a chronometer. You cannot rely on one for high precision or fine grained time, other than spending big bucks on purpose built, real time systems.

[Dan R Farrell]

Thanks for the responses. We did create a test program and are now running it at 10 ms in order to push things a bit and it is running at priority 55. We are using $SCHDWK with a repeat time value and are not calling it every cycle. It is now the only thing running on the Itanium box except for VMS, Decnet and TCP/IP. It is not part of a cluster. I guess my question is mainly that it does seem to work fine 99.99% of the time except for those 6 hour intervals. The synchronous nature of the event seems to indicate something else happening. I would expect more randomness from the event if it was related to any OS scheduling issue or something else also running at an elevated priority. We also created another test program using SETIMR and it does the same thing. I agree that if we really want guaranteed fixed 20 ms response we should probably use a hardware solution but we thought this would be good enough (and seemed to be in preliminary tests).

[Robert Gezelter]

Dan,

If I may put on my architecture hat and make a few observations.

I would not necessarily rush out for a hardware solution, but I would consider something in the nature of an IO driver for this type of task. OpenVMS time handling is subject to some imprecision, as John and others have noted. If something must be monitored precisely at a resolution that close to the precision of the system services, they are not appropriate.

I have seen this general genre of problem throughout my career, starting with second-generation PDP-11 systems. The answer is almost invariably the same: For high precision timing, get an external oscillator running at a significantly higher frequency, and have it interrupt every the counter gets to zero. At that point, use a device driver to perform the immediate actions and forward the summarized information to a process/task for more complete processing.

Since the time-critical portions of this code are in the driver's interrupt handling, little is likely to interfere with it.

For completeness, I note that just because one has not noticed an overhead operation lasting .02 second or so does not mean that they are not there. While cluster transitions and similar activities are well known, I would assume that there are other activities that can create similar situations. Jeff Schreisheim (formerly of the DECnet-11/RSX team) did a very nice article in Computer Design many years ago on why DECnet-RSX ended up implementing COMMEXEC, a special executive supplement to provide services needed by DECnet protocol modules. It makes very good reading even today.

- Bob Gezelter, http://www.rlgsc.com