Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Writing to disk using SYS$QIO

 
MButler
Occasional Visitor

Writing to disk using SYS$QIO

I'm trying to write to disk asynchronously. SYS$QIO seems like a good option. However, I'm having a very difficult time figuring out how to specify the file on the device that I wish to write to. From what I can tell in the documentation, argument p3 to SYS$QIO should be the "disk address"-- but how am I to get that? Any suggestions or examples?
23 REPLIES 23
Hein van den Heuvel
Honored Contributor

Re: Writing to disk using SYS$QIO

QIO to a disk is 'never' a good option.
QIO to a file on a disk is RARELY a good option. If it is 'that important' then perhaps you should be looking at $IO_PERFORM

Anywat, iF you want to go the all QIO route, then you need a QIO ACCESS first. See ACP section in DRIVERS reference manual.

More typically folks use RMS to open the file but set the UFO OPTION in the FAB. This will return a CHANNEL in the STV field which you can ue for a QIO(w).

HOWEVER... please consider your exact goals and skills. You may find that RMS SYS$WRITE / SYS$READ\, using the RMS AST / SYS$WAIT options is even more easy still.

AND, more than 1/2 th time I saw folks thinkg they needed QIO, they really just needed to learn more about RMS and WRTIE-BEHIND and/or DEFERRED-WRITE + SYS$FLUSH.

SO... if you need more help than this reply offers, then pleas ebe sure to be more clear about the problem you are trying to solve, and we may be able to help better.

Good luck!
Hein.
Hein van den Heuvel
Honored Contributor

Re: Writing to disk using SYS$QIO

Here is a tiny RMS - UFO - QIO example, for a specific QIO fucntion which is not directly avaibale in RMS. The powerful IO$_ERASE.

When playing with QIO, and VBN's, the $DUMP/BLOCK=(START:x,COUNT:y) will soon become your good friend.

Hein.



#include iodef
#include rms
#include stdio
#define FILE_SIZE 24
main()
{
int status, channel;
struct FAB fab;
short iosb[4];

fab = cc$rms_fab;
fab.fab$l_fna = "TMP.TMP";
fab.fab$b_fns = strlen(fab.fab$l_fna);
fab.fab$b_fac = FAB$M_GET | FAB$M_PUT;
fab.fab$l_fop = FAB$M_UFO;
fab.fab$l_alq = FILE_SIZE;

status=sys$create(&fab);
channel = fab.fab$l_stv;
if (status & 1) status = SYS$QIOW(0, channel, IO$_WRITEVBLK | IO$M_ERASE,
iosb, 0, 0, 0, FILE_SIZE*512, 1, 0, 0, 0);
if (status & 1) status = iosb[0];
if (status & 1) status = SYS$QIOW(0, channel, IO$_DEACCESS,
iosb, 0, 0, 0, 0, 0, 0, 0, 0);
return status;
}


Hein


Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

MButler,

I will heartily agree with Hein!

There is almost NEVER a need to do QIO to the disk. Asynchronous RMS access works quite well. I have been using it extensively for many, many years, particularly with other AST-based logic.

Is there a particular reason why QIO is being considered as opposed to RMS in an asynchronous fashion?

- Bob Gezelter, http://www.rlgsc.com
Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

MButler,

I should clarify something. A historical note is digression is appropriate.

When it was popular to run RSX-11 executables using the VAX-11 AME (still available if I recall correctly), it was a not infrequent happening to corrupt one's disk by "reassigning" IO from the terminal to the disk. Of course, TERMINAL QIOs generally had a zero P3 [hence the corruption: say goodbye to one's boot and home blocks].

Hence, the reason to recommend sticking with RMS. RMS will do the actual QIO operations with the XQP, gaining access to a particular file, and then do the QIO operations to access the data. This is far safer than writing one's own code in all but extremely rare occasions [ALMOST NEVER].

- Bob Gezelter, http://www.rlgsc.com
Hoff
Honored Contributor

Re: Writing to disk using SYS$QIO

The sys$qio works nicely, and IO$_WRITEVBLK is your best choice, and the usual approach here involves the Virtual Block Number (VBN), and the VBN of a file starts at one. The VBN runs from the first block of the file to the last block. And I'm here referring to disk blocks, and not to records within files.

The VBN pieces and using Virtual I/O gets around what the folks here are discussing; IO$_WRITELBLK and WRITEPBLK are (if you miss your intended block target) a good way to corrupt your disks. At its simplest, you perform a file-open and obtain a channel number, and then use whatever combination of READVBLK and WRITEVBLK operations and ASTs needed.

Hein's references to sys$read and sys$write and to the block-oriented operations in RMS are another way to avoid the risks of errant $qio operations.

If you're tossing full-on logical I/O, that's another discussion. And you'll need the Logical Block Number (LBN) for your target block. And writing to LBNs is at the programming level of rolling your own volume structure. (Yes, it's possible to do LBN writes entirely within an existing file structure, but that's, um, somewhat unusual.)

I've posted up a fairly generic RMS library for use from C applications.

It's been my experience that most decently-tuned RMS applications are reasonably fast. Yes, you can go faster, but that's usually more work than a quick pass through what RMS gives you.

Here's some reading material:
http://64.223.189.234/node/357
http://64.223.189.234/node/114
http://64.223.189.234/node/617

The pointer to the example C code download is in the above topics.

John Gillings
Honored Contributor

Re: Writing to disk using SYS$QIO

MButler,

Don't use $QIO. RMS services are a much better way to talk to disks. They'll be easier to use and perform significantly better than you will be able to achieve (remember, the RMS folk have had more than 30 years to get their code right!).

I would strongly urge you to modularise this data store. Consider the application view of the data and design an API that defines the set of required operations. You're then free to implement those operations independently of the application.
A crucible of informative mistakes
GuentherF
Trusted Contributor

Re: Writing to disk using SYS$QIO

"SYS$QIO seems like a good option"

Why do you think that is so? Did/do you know all the other options of accessing data on disk with OpenVMS?

/Guenther
Robert Brooks_1
Honored Contributor

Re: Writing to disk using SYS$QIO

I'm trying to write to disk asynchronously. SYS$QIO seems like a good option.

--

What's your language of choice? It's almost a certainty that using the language-specific I/O interface would suffice. Most DEC languages have DEC-specific extensions that allow for the use of some RMS features without resorting to using RMS directly.

If that not, then using the RMS services would almost certainly be easier/better/more maintainable than using QIO(W).

That one can do I/O via QIO doesn't make it the right thing to do. With proper privileges, one can do I/O by allocating their own IRP's and queueing them directly to a device UCB, but that doesn't make it a great idea, either.

-- Rob
Steven Schweda
Honored Contributor

Re: Writing to disk using SYS$QIO

If you get desperate for more disk QIO
example code, Info-ZIP UnZip uses some (in
[.vms]vms.c). UnZip version 6 (still not yet
released, but available in "BETA" form) even
includes support for ODS5 extended file
names. (I hope.)
Richard J Maher
Trusted Contributor

Re: Writing to disk using SYS$QIO

Hi MButler,

If all you want to do is Async I/O then I suggest that you stick with RMS and look at: -
http://h71000.www7.hp.com/doc/731final/4523/4523pro_001.html#asynch_completion

and: -
http://h71000.www7.hp.com/doc/731final/4523/4523pro_011.html#rmsref_rab_rop

If you're not fussed about things like locking and buffering and keyed-access, and want the raw performance and sheer grunt that can only be obtained by running 1% of the code-path then I also like $QIO! (After opening the file as Hein suggested UFO.)

Lastly, if you want VLM and Fast I/O and fancy being the next Oracle then don't miss my legendary "Big Buffs" example that can be found at: -

http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1203279

Cheers Richard Maher
MButler
Occasional Visitor

Re: Writing to disk using SYS$QIO

First off, thank you for the replies-- this is one of the most responsive forums I've come across.

Background: I'm a maintenance engineer for a legacy Fortran system that requires real-time response with a very high data volume. The tape drives are being decommissioned, so everything previously written to tape has to go to disk. SYS$QIO was the most obvious solution since that's what was used to write to tape.

I don't have internet access at work, so I'm printing off the posted references for later reading.

If I use SYS$CREATE with the UFO option and pass the returned channel to SYS$QIO, is disk corruption still a high risk?

Can SYS$CREATE handle logicals in the file name, or do I need to translate to the physical file name?
MButler
Occasional Visitor

Re: Writing to disk using SYS$QIO

Also-- I had come across SYS$CREATE in my slogging through of the user manuals today (before I had a chance to read the advice here), and experimented with it. I'm getting a "asynchronous operation not yet complete" error. Would this error come up for any asynchronous operation, or only I/O related operations? Just before the SYS$CREATE I was doing a SYS$DASSGN to clear the channel and a GETDVIW to check remaining disk space.

Do you know of any ways to find out what operation is blocking the SYS$CREATE? Or what I can rule out as possible culprits?
Steven Schweda
Honored Contributor

Re: Writing to disk using SYS$QIO

> [...] this is one of the most responsive
> forums I've come across.

That often happens when you ask scary
questions.

> [...] is disk corruption still a high risk?

Only if you try to write to it. If the block
you're writing to has been allocated to the
file, what could go wrong? If you go nuts
and start writing all over the place, then
you can cause trouble.

> Can SYS$CREATE handle logicals in the file
> name, [...]

Sure. It's still VMS.

> Also-- [...]

Actual code and actual error messages/status
values are often more helpful than vague
descriptions.
Hein van den Heuvel
Honored Contributor

Re: Writing to disk using SYS$QIO

> legacy Fortran system that requires real-time response with a very high data volume. The tape drives are being decommissioned, so everything previously written to tape has to go to disk.

Tape drives tend to be unshared write only.

That writing model fits perfectly to the standard RMS SYS$PUT (Fortrace WRITE) call with mulltple (3 - 10) large (64 - 127 block) buffers with WRITE BEHIND (RAB$V_WBH ... default for FORTRAN).

RMS will do all thr double buffering, asynchroneous stuff for you in that mode. Too easy.

>> SYS$QIO was the most obvious solution since that's what was used to write to tape.

It may have been obvious, but it was wrong.

>> I don't have internet access at work, so I'm printing off the posted references for later reading.

Did anyone tell them what century this is? Oh well. Their loss.

>> If I use SYS$CREATE with the UFO option and pass the returned channel to SYS$QIO, is disk corruption still a high risk?

Huh? Why would there ever be a risk of disk corruption?
Now 'File' corruption... maybe if you mis-manage VBN and LENGTHS
Or random 'Data' corruption if you grab unneeded privs and hack at the problem using LOGICAL BLOCK IO instead of Virtual block IO.

>> Can SYS$CREATE handle logicals in the file name, or do I need to translate to the physical file name?

That is precisely why you would want to use SYS$CREATE and not IO$_ACCESS | IO$M_ACCESS

If performance is a concern, be sure to toss in some useful values for ALQ and DEQ !

>> Also-- I had come across SYS$CREATE in my slogging through of the user manuals today (before I had a chance to read the advice here), and experimented with it. I'm getting a "asynchronous operation not yet complete"

So wait a while! Call SYS$WAIT.
But I suspect you fat-fingered the FAB$L_FOP option field and managerd to set the lower bit which requests an ASYNC operation, which darn well ought to return RMS$_PENDING.
Check out the 'subtle' distinction between FAB$V_xxx and FAB$M_xxx.
Or please just use the FORTRAN OPEN and let it do the work.


>>> error. Would this error come up for any asynchronous operation,

It is not an error . It is succes!

>> or only I/O related operations? Just before the SYS$CREATE I was doing a SYS$DASSGN to clear the channel

SYS$CREATE will grab a fresh channel, which may or might not relate to a channel you recently DEassigned from an, as far it the system is concerned, unrelated mission.

>> and a GETDVIW to check remaining disk space.

Noble, but silly. Just ask for what you need, and fail if you can not get it.

>> Do you know of any ways to find out what operation is blocking the SYS$CREATE?

Active IOs, doing the work you requested.

>> Or what I can rule out as possible culprits?

- Primary culprit : lack of understanding
- Secondary culprit : lack of understanding
- Third culprit: honest $M versus $V typo, but then with sufficient understanding you would have know that.. so back to cause #1.

Mostly teasing!

Cheers,
Hein.



Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

MButler,

Ok, now the picture comes into focus. My recommendation is to take one step back and focus on the real issue: performance.

Fact: The program is in FORTRAN.
Fact: Data volume is "high"
Unknown: What percentage of the CPU is used?
Unknown: What (precisely) does "high" mean [quantitatively]?
Unknown: What configuration are we speaking of?

Generally, in FORTRAN, the CPU cost is in formatting input and output. If the formatting can be bypassed, the overhead actually associated with RMS processing of an appropriately formatted file is very low.

What is the performance difference between tape and disk? Often, the answer is rooted in the fact that since tape is a sequential medium, not a shared, direct access medium, there is no need to control space allocation. Indeed, when doing a bandwidth intensive operation such as BACKUP, the overhead of the File Extend operations produce the most impact on performance.

File extension is expensive, and it is common to all disk usage, whether one uses RMS or bypasses it.

So, in the case described, what would I do as a first cut at solving the problem of changing writing to tape to writing to disk?

I would implement an interface common to all of my approaches, and implement it first using good RMS practices. If, as is likely, this approach yields 10,000% of the required performance (yes, I do mean a factor of 10 times the required performance), the need is satisfied without resorting to the low levels of using QIO directly.

First, I would abstract the interface to the output management, along the lines of CREATE_OUTPUT, WRITE_OUTPUT, and CLOSE_OUTPUT (while I was creating the interface library, I would also at a minimum define the corresponding READ routines). In your case, this all corresponds to a handful of subroutines and a named FORTRAN COMMON to maintain status.

Second, I would start by creating a standard RMS file, using the FORTRAN OPEN statement. I would specify a significant block size, blocking factor, and most importantly, file extension size and initial allocation. [Note: Hein and I have a friendly disagreement on whether blocking factors in excess of two are useful, the context of a rapidly growing file with extensions is one of the contexts where IMHO it is beneficial.]
Note that the working set and associated quotas may need to be increased to allow for the resulting buffer pool.

Third, Measure performance of the result. In most cases, this the combination of unformattted (or effectively unformatted) records, RMS buffer pool, large allocations, and large extents is very much more than sufficient to solve the problem.

The next step would be to very that all RMS settings that optimize output performance are correctly enabled (e.g., write-behind).

In almost all cases, that is sufficient. In any event, digging in and managing the file yourself using QIO (including manually extending the file, buffer pooling, etc.) are far, far more development effort. The approach that I am describing should require minimal time to implement and prove.

If performance is still an issue, more exotic approaches can be considered. For example, even volume shadowing (with a memory disk as one of the members of the shadow set) becomes a possible acceleration strategy. Bypassing RMS (as opposed to optimizing one's use of RMS) has far lower yield.

If performance is lacking, my first stop after the above would be directly using the file created using FORTRAN OPEN with RMS, bypassing the FORTRAN IO/formatting processing.

I hope that the above is helpful. If I have been unclear, please let me know.

- Bob Gezelter, http://www.rlgsc.com
Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

MButler,

Two small footnotes.

Depending on how old this code is, QIO may have been used because of limitations in ANSI tape support at the time the code was written. Those limitations do not apply to disk access today. The classic example is BACKUP, which does its extremely involved tape management. However, BACKUP uses RMS when writing a saveset to a file structured disk (which is why, one can do things such as place output savesets on different DECnet nodes using transparent file access, RMS is doing all of the actual work).

SYS$CREATE is RMS in effect. See Chapter 11 of the HP OpenVMS FORTRAN manual (the chapter is available online at
http://h71000.www7.hp.com/doc/82final/6443/6443pro_035.html#bottom_035 ; the entire manual is also available from the Documentation section of the OpenVMS www site at http://www.hp.com/go/openvms ).

This manual section specifically speaks of how to use RMS directly.

- Bob Gezelter, http://www.rlgsc.com
Hoff
Honored Contributor

Re: Writing to disk using SYS$QIO

Weird. I've always found (for basic block access) virtual $qio calls easy to use and easier to explain.

If you're used to ioctl-style access, that's $qio.

RMS is massively powerful and massively flexible, and the UI shows that; you can do exactly this block access with various of the RMS APIs, of course.

RMS is definitely the deep end of the programming pool, as is the XQP interface.

The most entertaining part of using $qio was the exact syntax of the file open syntax; getting RMS to pass through a channel number via the UFO path.
Mike Kier
Valued Contributor

Re: Writing to disk using SYS$QIO

Another very fast asynch I/O method is to map the (preallocated) file using $CRMPSC and use $UPDSEC to queue the writes.
Practice Random Acts of VMS Marketing
Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

Hoff,

I too do not find QIO problematic, however, many users whom I have encountered find dealing with the mechanics at that level difficult.

Since there is no proof on the table (at least at present) that anything more than buffered RMS is needed, I would not recommend going the QIO route.

I have seen "accidents" where someone goofed and used logical IO instead of virtual IO, combine that with a privileged user and ...

- Bob Gezelter, http://www.rlgsc.com
Hoff
Honored Contributor

Re: Writing to disk using SYS$QIO

I've posted up some RMS wrappers for C programmers previously, FWIW. Full source code with BSD-style licensing.

These days, RMS is further down my list of options and tools and I tend to go to something such as SQLite or MySQL or PostgreSQL or toward higher-level such as the C language APIs (or the above-mentioned wrappers); I do still use RMS for some applications, but I tend to go toward more portable and/or more modular I/O interfaces if and when the application allows that.

Younger programmers and experienced Windows and Unix programmers just don't know RMS. (I'm not going to argue that observation in any direction.) They do tend to know from block I/O and database I/O and language I/O.

But I'm not going to quibble with other preferences here, either.
Oswald Knoppers_1
Valued Contributor

Re: Writing to disk using SYS$QIO

"
Background: I'm a maintenance engineer for a legacy Fortran system that requires real-time response with a very high data volume. The tape drives are being decommissioned, so everything previously written to tape has to go to disk. SYS$QIO was the most obvious solution since that's what was used to write to tape.
"

Maybe you can use the LM driver here. This allows you to create a virtual tape unit on a disk file. No changes in your code required that way.

Oswald
Robert Gezelter
Honored Contributor

Re: Writing to disk using SYS$QIO

Mike,

>Another very fast asynch I/O method is to map the (preallocated)
>file using $CRMPSC and use $UPDSEC to queue the writes.

Indeed. However, one must be exercise caution. If all of the activity fits within the process' working set (or can be locked in physical memory), it is VERY fast. If actual paging of any kind takes place, the performance can experience "fits and starts" as page faulting occurs.

- Bob Gezelter, http://www.rlgsc.com

Jur van der Burg
Respected Contributor

Re: Writing to disk using SYS$QIO

>Maybe you can use the LM driver here. This allows you to create a virtual tape unit on a disk file. No changes in your code required that way.

Well, there's no such thing as LM driver. Technically it's LDdriver connected as device LM. V9.0 supports this, more info can be found on my website: http://www.digiater.nl/lddriver

Btw. I'm currenyly working on V9.2, due to be out in a couple of weeks. It will have variable (dynamic) container size support.

Fwiw,

Jur.