S/W Architecture Question

[Phillip Thayer]

I have a project that I am starting that requires me to pass data between a program running on a VAX and a program running on an AlphaServer.

The configuration we have is as follows:

VAX 4705A running VMS V6.2 clustered with an AlphaServer ES40 running VMS V7.3 with NICI interconnect over ethernet.

Cluster common queue manager and disk drives.

No memory channel devices.

Basically here is what I am supposed to accomplish. The AlphaServer is running our Attunity and connecting to all of our SQL databases outside of the VMS world. The VAX is running all of our legacy production applications and I cannot put Attunity on it due to the increase in CPU load that would create. I need to write a subroutine on the VAX that I can call in the legacy production code passing an SQL statement that will return the result set generated by the SQL statement. The SQL statement has to be executed on the the AlphaServer. Also, this subroutine will be called from numerous programs so it should be able to handle more than one SQL statement on the Alphaserver at a time.

What mechanism do I use to communicate between the VAX and the AlphaServer for this to happen?

My initial thought was to create a symbiont program and associate it to an execution queue on the AlphaServer. The subroutine on the VAX would take the SQL command passed it, create a temporary file containing the SQL statement and print it to the execution queue from the VAX. The symbiont program would take the file and create a detached process to execute the SQL and store the result set in a temporary file with the same file name but with a type of .RESULT and exit the detached process after deleteing the job from the queue. The subroutine on the VAX would then after printing the file to the queue loop through a LIB$FIND_FILE looking for the .RESULT temporary file for the SQL result set.

Is this making it more difficult than is necessary? Is there an easier way to do this? Any help on this would be greate.

Thanks,
Phillip Thayer

[Peter Quodling]

A lot depends on what you are trying to achieve with respect to performance, resilience, ease of programming etc.

If this being "hit hard" ie lots of requests? Is in-order execution a consideration. Perhaps Bea MQ, or IBM MQSeries is appropriate? Is it an absolutely must happen scenario? (Our old friend RTR)

Also read up on ICC in the VMS doc kit...

[Robert Gezelter · Accepted solution]

Phillip,

Having done this sort of thing many, many times, I normally recommend the use of DECnet logical links from an API implemented as a shareable library on the VAX side to a network server on the Alpha.

There are several reasons why I recommend the use of DECnet logical links for situations of this type:
- they provide for the possibility of authentication and access control
- they provide a controlled situation if the requesting process fails during the execution of the request
- they provide a controlled situation in the event that the server malfunctions while processing the request
- they permit the question of whether the server should be single threaded or multi-threaded to be an implementation or system management question, not a feature of the API
- the cost of the communication, even within a node, is minimal

As I said, I have done this many times, even in situations where we had rather unstable applications software using the API, and I never had a problem in the server induced by the application.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Phillip,

My apologies, but I realized something was omitted from my post after hitting the SUBMIT button.

The use of a DECnet logical link also makes the requestor's CPU architecture. If applications presently on the VAX move to the Alpha, or indeed to Integrity, the interface remains stable and useable.

- Bob Gezelter, http://www.rlgsc.com

[Antoniov.]

Philip,
with native vms software the best practice is DecNet. I'm with Robert for same reasons. I use DecNet to synchronize software between vms machines.
However, you have to try, because DecNet might be slow used in SQL statement.
Another way may be tcp/ip services.

Antonio Vigliotti

[Robert Gezelter]

Antonio,

The speed question always surfaces, and it is easily addressed.

Locally, within a node, the message rate is impressive (on a very old Alpha, tens of thousands of messages/second). Between nodes, it is a question of underlying network media.

The key is avoiding the common mistake: opening/closing the network connection for each query. This is as deadly to achieving performance as opening/closing/repositioning a file on reading every record. However, I have seen it done often.

That is one of the reasons that I mentioned hiding the network issues in a shareable library. That allows the management of the network issues to be beyond the knowledge (and control) of the application.

- Bob Gezelter, http://www.rlgsc.com

[Ruslan R. Laishev]

Phillip,
is the amout of data to exchange between nodes is not very big ( < 2k), I suggest to use TCP/IP and UDP, because it's will allow to realise a multistreaming sending/receiving of SQL statements and results. Be advised that ICC give not some important options for implementing a bullet-prof processing, DecNet - cool thing, but in the same time it need to spread a primary configuration on the NCP database and so on...

With UDP/IP you will need to implement a simple authentication procedure and timeout/retry processing, but you will get advantages of controlling your applications.

An exchange with files+symbiont - a huge overhead of processing, don't do this. :-)

Just my 2 cents.

[Willem Grooters]

@Ruslan:

DECNet - despite the extra setup - has great advantages above IP. In a VMS-to-VMS connection, when set up with redundancy, loss of a path will take aover automatically - _without_interrupting_the_connetion_. With IP, you loose it. With UDP, you will have to check if the message received is in-order and complete, with TCP that's not needed.
Setting up a basic DECNet network requires somewaht extra work, but you can only win compared to IP (see above).

Within the cluster: have you considered network mailboxes? On the alpha, create a detached process that will create a _temporrary_ mailbox, to receive the SQL-statements from the VAX and process them; Set a write-attention AST on that channel and hibernate. The AST will wake up the process, after which it will read the mailbox and process the message.
The process on the VAX will create a (temporary) mailbox for the Alpha-process to write the results to. A SQL-statement is sent to the mailbox owned by the process on the Alpha, and results are read from the other mailbox. If size matyters, youy _could_ use a IP socket for the results ;-)
Great advantage: the process on the VAX will know if the server on the Alpha is running, if the channel is assigned before, and de-assigned after use (on the VAX side)
Of courcxse, if a result is BIG - you may pass the resulting fiulename via the return-mailbox.

Willem

[Ruslan R. Laishev]

Willem,
VAX can be get out from the cluster because it's a reason of performance degrataion of hole cluster.

IP-aliasing (or using pseudo-interface) and connection-state-less mature of UDP allows to implement failover and so on... Yes, reordering is the problem which can be resolved by packet numeration at app-level.

For failovering on DECNet it need to setup DECnet cluster-alias, another useful thing is the load balancing with user-written procedure for calculation of "load factor".

[Wim Van den Wyngaert]

Is there a reason why you can't use the client software of the database on alpha ?
This way you write a subroutine that runs on alpha and accesses the database on vax or whatever system it is on (using decnet or IP). You only have to setup sqlnet (oracle) or an interfaces file (sybase).

Wim

[Phillip Thayer]

Greate responses from everyone. Thanks. First let me say I assigned points for everyone.

The thought of using RTR/MQSeries/BEA MQ is really out of the question. We are on a very tight budget and thrid-party software is out of the question. This is something I will have to write myself.

TCP/IP is not something that I want to use simply because the only programming language choice I have is VMS Basic and it is not very easy to do TCPIP with it. I have written some TCPIP code to handle requests to ASP pages with responses from them and that was difficult to accomplish for such a simple task.

I have not been programming for a while and my skills are a bit rusty so I completely spaced on the DEecnet Locical links communication idea. I have used this in the distant past for some very resilient applications dealing with data base updating between nodes. This is probably the solution I am looking for. Then I can structure it to be multithreaded as well.

There was one mention of MBX usage between nodes. I am not familiar with the mechanism for creating mailboxes on one member of a cluster and being able to access them from another member of a cluster. Would there be some DECNet involved in the underlying VMS communications here? How would that work? Is it an easier way of doing the DECNet Logical Links?

I really am extremely satisfied with the responses so far.

Phillip

[Ian Miller.]

Phillip, have you read
http://h71000.www7.hp.com/doc/82FINAL/5841/5841pro_006.html#chap_process_communication
for an overview of various ways processes
can commnicate. Personally I would use DECnet or the SYS$ICC* routines.

[Phillip Thayer]

Ian,

I am not sure but I believe that the SYS$ICC* routines are not part of OpenVMS V6.2. Since my VAX is running V6.2 I would not be able to use the SYS$ICC* routines. That would mean the DECNet solution is probably what I am going to have to use.

Phil

[Robert Gezelter]

Phillip,

The intra-cluster mailbox idea does not use DECnet, it would, if I remember correctly, use SCS to transfer the data between the nodes.

Generally, even within a node, I always recommend the DECnet logical link. There are only two programming issues that occur when using links from BASIC:

- it helps significantly to use the record definition extensions to define the formats of the messages (insofar as the formats are not already in the VMS-supplied libraries, which most of them are).

- it also helps to write the code as a set of AST routines with a vestigial main program. The BASIC run time environment, like the FORTRAN run time environment, is not re-entrant. However, if you have an essentially non-existent main program (e.g., little more than a WHILE loop surrounding a call to SYS$HIBER, see my presentation on ASTs at my www site http://www.rlgsc.com/presentations.html; I will post the precise URL later, as the site is experiencing conneectivity problems apparently due to yesterday's severe storm).

The reason for my preferenc for the DECnet approach, over the mailbox approach is the degree of support provided by the system:

- if connectivity is lost, DECnet will notify both client and server processes immediately
- if either server OR client exits unexpectedly, the other process will be notified automatically.
- everything can be accomplished from an account with no privileges beyond NETMBX and TMPMBX, thus no danger of compromising security and integrity (pun not intended) of the underlying system.

Mailboxes would not inherantly give notification of loss of connectivity nor or partner exit. A TCP connection has very long timeouts, and little provision for writing servers without at least the administrative priveleges to make them accessible to the inbound request. UDP, by definition, does not provide guarantees of sequentiality nor for that matter, guaranteed delivery, which creates many problems.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[Ian Miller.]

I think you are correct that SYS$ICC* are not present in VAX/VMS V6.2.

[Robert Gezelter]

Phillip,

If your "budget is tight" and your "programming skills are rusty", I strongly recommend you ignore multithreading, at least during the initial version of the project.

Architecturally, multithreading should be invisible to the client process and its supporting library. It merely increases the complexity of the code that you write, often not giving a significant performance boost.

A better first approximation is to implemement the single threaded DECnet-based applications service, and then INSTALL it as a known, shareable image. This means that the read only data and code pages of the image will be shared between the different server processes, a significant savings of code space.

Single threading also reduces the potential for an error induced from a single client's request can have an impact on other users.

- Bob Gezelter, http://www.rlgsc.com

[Phillip Thayer]

That sounds like a good plan. I will do the sigle-threaded app first and make sure I do it in a way that will be easily upgradeably to a multithreaded version.

Thanks for all the help. I have a good idea of what I need to do and will be starting on the design this week. Once I start writing code if I have any problems I will post questions here. This has been a terrific source of help.

Thanks,
Phil

[Ruslan R. Laishev]

Phillip,
I don't know exactly what "SQL-server" you meant, anyway have a look to FreeTDS library/ It's a small library to "talk" with Sybase and M$ SQL server...


http://starlet.deltatel.ru/~laishev/work/freetds/

[Phillip Thayer]

The way our system is setup is we have a MS Windows box running SQL Server and Attunity Connect with several databases defined on it. On our AlphaServer we have Attunity Connect running with a datasource setup that contains definitions for all of our RMS data files (about 250 different files). The SQL Server on the MS Windows box has a database configured using a Linked Server to be able to have access to all the RMS datafiles defined on the AlphaServer Attunity datasource. This allows us to be able to use SQL from outside the VMS world to query the RMS data on the VMS systems

What I am attempting to accomplish now is for the VMS world to be able to access the SQL databases on the MS Windows box by using Attunity to access the SQL databases via the AlphaServer.

Ultimately, we want to be able to use the SQL access for all files regardless of where they reside (VMS or Windows) for all of out legacy applications via a standard library of routines on the VAX. This will enable us to move the files where we want with little effort other than redefining the datasources in Attunity.

Phillip

[Ruslan R. Laishev]

Philip,

FreeTDS - is the your choice! :-) You can directly from the VAX or Alpha doing what you want with data resides in the M$ SQL box.

I think so...

[Ruslan R. Laishev]

Home page of the project:

www.freetds.org

[Antoniov.]

Phillip,
something may be simple for somebody while is complex for some other.
Like you, my skill is bit rusty so I use always same code :-)
If you know $QIO service, you can use tcp/ip without programming socket; you can just enable SRI QIO interface in TCPIP$CONFIG (oi UCX$CONFIG on vax).
However, if mailboxes work in cluster, using of mailbox is the simplest solution. You can program mailboxes like you open, read and write a file.

Antonio Vigliotti