Writing to disk using SYS$QIO

[Richard J Maher]

Hi MButler,

If all you want to do is Async I/O then I suggest that you stick with RMS and look at: -
http://h71000.www7.hp.com/doc/731final/4523/4523pro_001.html#asynch_completion

and: -
http://h71000.www7.hp.com/doc/731final/4523/4523pro_011.html#rmsref_rab_rop

If you're not fussed about things like locking and buffering and keyed-access, and want the raw performance and sheer grunt that can only be obtained by running 1% of the code-path then I also like $QIO! (After opening the file as Hein suggested UFO.)

Lastly, if you want VLM and Fast I/O and fancy being the next Oracle then don't miss my legendary "Big Buffs" example that can be found at: -

http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1203279

Cheers Richard Maher

[MButler]

First off, thank you for the replies-- this is one of the most responsive forums I've come across.

Background: I'm a maintenance engineer for a legacy Fortran system that requires real-time response with a very high data volume. The tape drives are being decommissioned, so everything previously written to tape has to go to disk. SYS$QIO was the most obvious solution since that's what was used to write to tape.

I don't have internet access at work, so I'm printing off the posted references for later reading.

If I use SYS$CREATE with the UFO option and pass the returned channel to SYS$QIO, is disk corruption still a high risk?

Can SYS$CREATE handle logicals in the file name, or do I need to translate to the physical file name?

[MButler]

Also-- I had come across SYS$CREATE in my slogging through of the user manuals today (before I had a chance to read the advice here), and experimented with it. I'm getting a "asynchronous operation not yet complete" error. Would this error come up for any asynchronous operation, or only I/O related operations? Just before the SYS$CREATE I was doing a SYS$DASSGN to clear the channel and a GETDVIW to check remaining disk space.

Do you know of any ways to find out what operation is blocking the SYS$CREATE? Or what I can rule out as possible culprits?

[Steven Schweda]

> [...] this is one of the most responsive
> forums I've come across.

That often happens when you ask scary
questions.

> [...] is disk corruption still a high risk?

Only if you try to write to it. If the block
you're writing to has been allocated to the
file, what could go wrong? If you go nuts
and start writing all over the place, then
you can cause trouble.

> Can SYS$CREATE handle logicals in the file
> name, [...]

Sure. It's still VMS.

> Also-- [...]

Actual code and actual error messages/status
values are often more helpful than vague
descriptions.

[Hein van den Heuvel]

> legacy Fortran system that requires real-time response with a very high data volume. The tape drives are being decommissioned, so everything previously written to tape has to go to disk.

Tape drives tend to be unshared write only.

That writing model fits perfectly to the standard RMS SYS$PUT (Fortrace WRITE) call with mulltple (3 - 10) large (64 - 127 block) buffers with WRITE BEHIND (RAB$V_WBH ... default for FORTRAN).

RMS will do all thr double buffering, asynchroneous stuff for you in that mode. Too easy.

>> SYS$QIO was the most obvious solution since that's what was used to write to tape.

It may have been obvious, but it was wrong.

>> I don't have internet access at work, so I'm printing off the posted references for later reading.

Did anyone tell them what century this is? Oh well. Their loss.

>> If I use SYS$CREATE with the UFO option and pass the returned channel to SYS$QIO, is disk corruption still a high risk?

Huh? Why would there ever be a risk of disk corruption?
Now 'File' corruption... maybe if you mis-manage VBN and LENGTHS
Or random 'Data' corruption if you grab unneeded privs and hack at the problem using LOGICAL BLOCK IO instead of Virtual block IO.

>> Can SYS$CREATE handle logicals in the file name, or do I need to translate to the physical file name?

That is precisely why you would want to use SYS$CREATE and not IO$_ACCESS | IO$M_ACCESS

If performance is a concern, be sure to toss in some useful values for ALQ and DEQ !

>> Also-- I had come across SYS$CREATE in my slogging through of the user manuals today (before I had a chance to read the advice here), and experimented with it. I'm getting a "asynchronous operation not yet complete"

So wait a while! Call SYS$WAIT.
But I suspect you fat-fingered the FAB$L_FOP option field and managerd to set the lower bit which requests an ASYNC operation, which darn well ought to return RMS$_PENDING.
Check out the 'subtle' distinction between FAB$V_xxx and FAB$M_xxx.
Or please just use the FORTRAN OPEN and let it do the work.


>>> error. Would this error come up for any asynchronous operation,

It is not an error . It is succes!

>> or only I/O related operations? Just before the SYS$CREATE I was doing a SYS$DASSGN to clear the channel

SYS$CREATE will grab a fresh channel, which may or might not relate to a channel you recently DEassigned from an, as far it the system is concerned, unrelated mission.

>> and a GETDVIW to check remaining disk space.

Noble, but silly. Just ask for what you need, and fail if you can not get it.

>> Do you know of any ways to find out what operation is blocking the SYS$CREATE?

Active IOs, doing the work you requested.

>> Or what I can rule out as possible culprits?

- Primary culprit : lack of understanding
- Secondary culprit : lack of understanding
- Third culprit: honest $M versus $V typo, but then with sufficient understanding you would have know that.. so back to cause #1.

Mostly teasing!

Cheers,
Hein.

[Robert Gezelter]

MButler,

Ok, now the picture comes into focus. My recommendation is to take one step back and focus on the real issue: performance.

Fact: The program is in FORTRAN.
Fact: Data volume is "high"
Unknown: What percentage of the CPU is used?
Unknown: What (precisely) does "high" mean [quantitatively]?
Unknown: What configuration are we speaking of?

Generally, in FORTRAN, the CPU cost is in formatting input and output. If the formatting can be bypassed, the overhead actually associated with RMS processing of an appropriately formatted file is very low.

What is the performance difference between tape and disk? Often, the answer is rooted in the fact that since tape is a sequential medium, not a shared, direct access medium, there is no need to control space allocation. Indeed, when doing a bandwidth intensive operation such as BACKUP, the overhead of the File Extend operations produce the most impact on performance.

File extension is expensive, and it is common to all disk usage, whether one uses RMS or bypasses it.

So, in the case described, what would I do as a first cut at solving the problem of changing writing to tape to writing to disk?

I would implement an interface common to all of my approaches, and implement it first using good RMS practices. If, as is likely, this approach yields 10,000% of the required performance (yes, I do mean a factor of 10 times the required performance), the need is satisfied without resorting to the low levels of using QIO directly.

First, I would abstract the interface to the output management, along the lines of CREATE_OUTPUT, WRITE_OUTPUT, and CLOSE_OUTPUT (while I was creating the interface library, I would also at a minimum define the corresponding READ routines). In your case, this all corresponds to a handful of subroutines and a named FORTRAN COMMON to maintain status.

Second, I would start by creating a standard RMS file, using the FORTRAN OPEN statement. I would specify a significant block size, blocking factor, and most importantly, file extension size and initial allocation. [Note: Hein and I have a friendly disagreement on whether blocking factors in excess of two are useful, the context of a rapidly growing file with extensions is one of the contexts where IMHO it is beneficial.]
Note that the working set and associated quotas may need to be increased to allow for the resulting buffer pool.

Third, Measure performance of the result. In most cases, this the combination of unformattted (or effectively unformatted) records, RMS buffer pool, large allocations, and large extents is very much more than sufficient to solve the problem.

The next step would be to very that all RMS settings that optimize output performance are correctly enabled (e.g., write-behind).

In almost all cases, that is sufficient. In any event, digging in and managing the file yourself using QIO (including manually extending the file, buffer pooling, etc.) are far, far more development effort. The approach that I am describing should require minimal time to implement and prove.

If performance is still an issue, more exotic approaches can be considered. For example, even volume shadowing (with a memory disk as one of the members of the shadow set) becomes a possible acceleration strategy. Bypassing RMS (as opposed to optimizing one's use of RMS) has far lower yield.

If performance is lacking, my first stop after the above would be directly using the file created using FORTRAN OPEN with RMS, bypassing the FORTRAN IO/formatting processing.

I hope that the above is helpful. If I have been unclear, please let me know.

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

MButler,

Two small footnotes.

Depending on how old this code is, QIO may have been used because of limitations in ANSI tape support at the time the code was written. Those limitations do not apply to disk access today. The classic example is BACKUP, which does its extremely involved tape management. However, BACKUP uses RMS when writing a saveset to a file structured disk (which is why, one can do things such as place output savesets on different DECnet nodes using transparent file access, RMS is doing all of the actual work).

SYS$CREATE is RMS in effect. See Chapter 11 of the HP OpenVMS FORTRAN manual (the chapter is available online at
http://h71000.www7.hp.com/doc/82final/6443/6443pro_035.html#bottom_035 ; the entire manual is also available from the Documentation section of the OpenVMS www site at http://www.hp.com/go/openvms ).

This manual section specifically speaks of how to use RMS directly.

- Bob Gezelter, http://www.rlgsc.com

[Hoff]

Weird. I've always found (for basic block access) virtual $qio calls easy to use and easier to explain.

If you're used to ioctl-style access, that's $qio.

RMS is massively powerful and massively flexible, and the UI shows that; you can do exactly this block access with various of the RMS APIs, of course.

RMS is definitely the deep end of the programming pool, as is the XQP interface.

The most entertaining part of using $qio was the exact syntax of the file open syntax; getting RMS to pass through a channel number via the UFO path.

[Mike Kier]

Another very fast asynch I/O method is to map the (preallocated) file using $CRMPSC and use $UPDSEC to queue the writes.

[Robert Gezelter]

Hoff,

I too do not find QIO problematic, however, many users whom I have encountered find dealing with the mechanics at that level difficult.

Since there is no proof on the table (at least at present) that anything more than buffered RMS is needed, I would not recommend going the QIO route.

I have seen "accidents" where someone goofed and used logical IO instead of virtual IO, combine that with a privileged user and ...

- Bob Gezelter, http://www.rlgsc.com