Queue Load Balancing In The Modern World

[Robert Atkinson]

Back in the days of LN03's and the like, the VMS print symbionts had a direct connection to the print engine and could tell what the printer was actually up to. This enabled it to know if the printer was busy or not, so load balancing was easy.

In the modern world, where COLD text is printed, manipulated and then sent to a Windows queue, this doesn't really work so well.

My question (finally got there) is how do others cope with this tricky problem. I'm going to invoke some load balancing techniques within DCL, but it would be much better if VMS knew what a printer was doing again. Does DCPS have anything in it that can do this. Is there any way to manually interrogate the printer status and feed this back to the symbiont?

Thanks, Robert.

[Bob Blunt]

Robert, some background info would be helpful...  OpenVMS (or VMS) version, printer types, queue setup, DCPS version you're using, etc.

DCPS depends on full bi-directional communication for supported printers to work best and driving printers from another O/S muddies these waters.  In general, however, you're mostly right if you're spooling to printers that are controlled by some other environment.  There are products that can be purchased from other vendors that might work better in a mixed printer world.  In my opinion, though, if you're trying to keep that control on your VMS systems then you need to focus on using printers that more closely resemble "The Good Ole Days."  Some still exist and you and your company will have to decide if they fit your needs or if you're stuck trying to fit an OpenVMS peg into a Windows hole.  Put another way?  I feel your results will vary depending on the printers (and their setup) that you use.  Most current PC-friendly printers that I'm experiencing don't seem to  have readily available Postscript (or other printer language) interpreters.  And since VMS doesn't have the native ability to generate direct printer control for some of those units (with or without DCPS) most of the burden of control has to be given to the PC world.

Even in an office with a variety of printer types our results were not always predictable.  I had setup direct queues to HP LaserJets, HP Color LaserJets and, for a short time, a few remaining DEC printers.  Although I had made sure that our local printers had both Ethernet interfaces and PostScript interpreters those printers didn't always work as expected.  I never was able to get output from one of our MFPs with Ethernet but the DEC printers were more predictable.  The HP laser printers seemed to work better with Windows but we lacked any tie-in software that allowed us to get better results from VMS printing.

Focusing on the printers that are listed as supported by recent releases of DCPS can help some but I can't remember if that resource gives explicit requirements for connectivity from VMS TO those printers.  I would expect that they'd work best if setup as directly as possible.  But if you're stuck with sharing printers spooled and controlled by another O/S then you might look around for other software that might make it better.  GrayMatter and Northlake are two vendors that come to mind.

[Hoff]

As an incremental step, consider using network-based printers (and get the intermediate boxes out of the way), or use something akin to the EXECSYMB package and roll your own queue-management logic.    

To move past this design, you may (will?) have to figure out how to connect into the Windows box in parallel to and asynchronous to the printing activity, and particularly determine when the Windows box is done with its processing, and use this to set up a customized print symbiont or analogous "back-pressure" for that task.

Moving toward a more recent solution, you're running what amounts to a compute grid and not "just" some printer queues, and you'll be locating and porting over some grid-scheduling software, or (less desirably) rolling your own grid software.  VMS, unfortunately, doesn't provide clients for modern grids, and comparatively few (if any?) of the tools that manage compute clusters ("clusters", but not in the sense that VMS uses) have been ported over to VMS.  And this case has the added complexity of the Windows boxes, and any solution will necessarily have to involve communications with those.

In an overview of a typical grid-like design, you'd have VMS boxes producing "stuff" and (for this part) the Windows boxes waiting for and consuming and processing the "stuff" for printing.  The Windows boxes would then push the jobs to the printers, or advertise that they've now produced some "stuff" and whatever is driving the printers is waiting to fetch and process that.  Whatever's running your grid would be passing around the "stuff" events, and the requests for "stuff".

The underpinnings of this might be based on AMQP...

The boxes I'm usually dealing with (well, other than the VMS boxes) are typically configured as members of grids, and (for instance) source code compilations drop down the "rabbit hole" and are processed on compute servers on the network.  For these particular builds, this is based on the distcc tool, though there are other options.  For other stuff, Xgrid.  And there are a number of other tools and protocols that can schedule jobs.

This is fairly old technology, all things considered. (It's past what VMS typically offers, but it's commonly found on other platforms.)  But yes, something is going to have to manage that herd of VMS and Windows boxes here.

[Robert Atkinson]

I guess my hope was that there was at least a documented API within the symbiont that could be hooked into, where I could force the status of the remote printer. I did think about ExecSymb, but I'm a little wary about basing such a critical process on Freeware.

The Engineering guys are in London next week for the UK TUD, so perhaps I'll throw this one in there direction and watch their faces :)

Thanks, Rob.

[Mike Kier]

I used EXECSYMB in a production environment at two different employers - one for printing prescription labels on an image-based Zebra thermal printer and another to perform an FTP file transfer and send an email link to the document - I would have no hesitation using it in any production environment - the sources are available and the VMS symbion APIs are well documented.

[Hoff]

There has traditionally been no good facility for modifying an already-modified symbiont.  

There are discussions of the PSM-related pieces around, and varying success at modifying existing symbionts.  And AFAIK, there's no way into the existing DCPS symbiont pieces documented, nor particularly feasible.

I'd also suggest a recalbration around your view of Freeware code, too,  That's increasingly what you'll be working with and depending on, going forward, whether within your environment, or with tools such as Apache, witjhin VMS, and otherwise.

A server queue would probably be fairly close to what you want, where that queries the printers, and dispatches the jobs appropriately.  But you're back-dooring your way into grid management and distributed scheduling, so I'd suggest going into this project with your eyes toward the larger environment.  (Not intending to solve it all at once certainly, but rather to avoid implementations and decisions that would constrain your environment later.)

[Robert Atkinson]

I've always been an avid supporter of Freeware, and still use a number of utilties on my Production systems. The problems generally ocurr on code written or ported by individuals rather than communities.

To date, I've had to stop running at least 3 Freeware utilites because they've broken during migration or VMS upgrades (and still have a fatal problem with HGFTP on my Dev node). I've made numerous attempts to contact the authors to no avail, so have had to write work-arounds. ZMENU was one of those utilies that we had plumbed into our Production systems, so it gave us quite a headache when we couldn't use it any more.

For these reasons, I have to be very wary over what I introduce, especially when I have no way of backing it out by writing my own code.

Rob.

[Hoff]

Other options to backing out the Freeware packages (or alternatives to refraining from incorporating the packages) include having the package brought in-house for maintenance, or a contract to have the package fixed, too.

FWIW, a number of the folks around will make those same calculation around the long-term viability of the underlying operating system and the underlying hardware platform.  The availability, maintenance and quality of the freeware, shareware and applications for the platform feed into that calculation and that decision.  The associated decisions might then involve (for instance) a port from OpenVMS Alpha to OpenVMS I64 or OpenVMS on HPVM, or a port from OpenVMS to Linux on ProLiant, or otherwise.  (I'm serving applications and requirements that once consumed whole OpenVMS boxes off of toaster-sized x86 boxes; the hardware and the software technology both continue to change.)

Some awareness of the effects of vendor (and of freeware, for that matter) product lifecycles and the increasing rate of platform changes is nearly necessity in this industry.   You're looking at the Freeware here, but what of the platform?  While VMS has been quite staid and reliable in that regard; I'm getting utterly slammed with new features updates and enhancements on the other platforms I deal with; changes and updates and features that are of direct value to the business.

On the subject of ZMenu front-end interfaces, packages such as Django completely blow away many of the front-ends implementation tools I've worked with, as an example.  This in terms of the speed and flexibility of the development, the features, lower maintenance costs, etc.  Though I've been using it elsewhere, I haven't tried Django on VMS; I'd expect that to work.

Given that none of this is going to be addressed much less resolved here, I'd take a few steps back when the next opportunity arises, and ask yourself (and with the help of your management and your user base) where you want to end up here, and how you'll get there, and what the budget and staffing and purchase and acqusition requirements might be, and what parts of the environment might be retired.  Make time for this planning, if you've not already done so.

[Robert Atkinson]

Given your comments and going back to my original problem about Load Balancing, this is a very tiny cog in a large wheel, but unfortunately a critical one.

I'm sure that if I started discussing bringing in outside resource to resolve the problem (i.e. a C developer), I be very quickly told that I'm going about it in the wrong way, which is perhaps true. There are clearly many ways to crack this nut, including a full blown off-VMS queue managment solution, but I'm not willing to get into those complexities to achieve this end.

There are many ways VMS _could_ be enhanced such as this addition to the print symbiont, but as you seem to be hinting to, other OS's are pushing forward constantly where VMS seems to be slowing down more and more. At least that's what I feel. Still, I'll continue to get 110% out of it until I'm told otherwise.

Thanks, Rob.