About IO_PERFORM

[Bob CI]

In addition on my application improvement with my previous post about POSIX THREADS ( do you remember ? ), i´m trying to improve the file's writings.

Initially, we use FORTRAN for file accesing, then i changed to RMS and the times were better.

But i need to improve this times. I founded examples with FASTIO. The example in sys$examples are very easy form my needs. I want to write with FASTIO in a relative file. I achieve to read relative files ( when i read the file with READVBLK, i detect that the first block isn´t data, but the second, the third and next ones are ).

But when i´m trying to write, the only i achieve is to corrupt the file. First, i thought "i haven´t to write the first block, cause is the file header ", but if i write the second block, i have the same problem

do I need more information that i haven´t when i was writing with FASTIO ?? ( files structures at low level ),etc... Can i achieve more complex examples with FASTIO writings.

Thanks in advance..

[Hein van den Heuvel]

IO_perfrom can reduce the CPU overhead for IOs. Nothing else.
It will NOT speed up the IO itself.

By avoiding RMS and using QIO you possibly reduced that CPU overhead already by 90% (just the overhead)... if the application did not exploit things like the RMS buffer pool and RMS file sharing and so on.

IO_perform, when done properly, may save you an other 20% of the 10% that's left.
I fail to believe that you can notice / measure this, since you even failed to indicate/understand what the current bottleneck might be.

RMS relative files are trivial to understand, and often poorly used.

Does the applicaiton use the notion of 'deleted' or 'valid' records?

Is a useful bucketsize chosen?

Is a reasonable record size chosen, based on needs, and not on looks like the often found, but extremely stupid 512 byte records?

Read: http://h71000.www7.hp.com/doc/731FINAL/4506/4506pro_005.html#020_relativefileorganization

Do: ANALYZE/RMS/INTERACTIVE

PLEASE, do yourself a favor and start at the beginning and do not jump at a sexy sounding solution (threads, io_perform, virtualization :-).

What is the application trying to accomplish?
What are the constraints? (sharing?)
What are the rough performance requirements?
Where is the current bottleneck?.. how do you know?

Get professional help!

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Robert Brooks_1]

What, exactly, are you trying to do? If you are trying to emulate the Files-11 XQP (you make reference to a file header), you are almost certainly going to get it wrong.

I agree with Hein; I'd take a step back, breathe deeply, and try to articulate what you'd like to accomplish. There are likely better solutions than the one you're now attempting to implement.

While it's not likely relevant to your problem, there have been some fixes for $IO_PERFORM, so please make sure you're up-to-date with your patch kits. I believe the problems addresses were I/O hangs.

-- Rob

[Jess Goodman]

Relative files, indexed files, and sequential files with variable-length records all have RMS meta-data mixed in with the user data. If you attempt to write to them using QIOs or FASTIOs you will very likely corrupt the file.

If you really want to bypass RMS you should use a sequential file. If you need random access to the records you should use fixed record size format; otherwise you can could use one of the stream record formats.

If the file will also be accessed via RMS at the same time as a program that bypasses RMS (or even by two non-RMS programs), there are a couple of issues you must consider.

First when you bypass RMS you also bypass RMS record-locking so if you re-write a record without RMS you might lose an update made to another record by another process if it's in the same buffer.

You also bypass RMS buffers which are also coordinated using the distributed lock manager. So if a program was using RMS to re-read a record waiting for an update, it may never notice an update made by a non-RMS program.

Bypassing RMS for speed could end up being counter productive. We wrote programs that read and wrote data files using $QIOs way back in our PDP-11 RSX11M days and then ported them to a VAX. At first we were happy with how fast they ran.

But time went by and our VMS systems got faster, but these programs ran just as slow as before. This was because of the $qio buffer size hard-coded into the programs was way too small for these bigger systems.

Others programs that did use RMS ran faster on VMS and faster still after a simple SET RMS command to increase their RMS buffer size.

Having said all that I should add that we have found that we can speed up programs considerably by turning off RMS record locking in cases where we are sure that it will not be needed. This can be done without bypassing RMS by setting the FAB$M_UPI bit in the FAB$B_SHR field.

We do this for a many of our sequential files that are read by many different processes but are only ever written to by one program at a time (which the writer program enforces by setting field FAB$B_SHR to FAB$M_SHRGET). The writer program must also flush RMS buffers regularly, and my caveat about re-reading records applies.

[Hoff]

No.

Back up.

Define the problem.

Find the performance bottleneck.

Characterize the performance limits.

I don't yet know what hardware is involved here. And in any case and in any configuration, an application is not going to go faster than what the hardware permits, short of a hardware upgrade, or a radical re-think on how the data is stored and accessed.

Without a case for this implementation well beyond what I've read on this issue to date, I'd certainly counsel (strongly) against use of sys$qio(w) and sys$io_perform(w). (And against threads.) These approaches are complex and fussy, and whether or not the effort involved will help improve performance is certainly not yet known.

If your current solution provides most of the practical bandwidth of your disk I/O system and your system hardware, for instance, you'll be wasting your skills and your time and your effort with a migration over to sys$io_perform(w).

[Richard J Maher]

Hi Roberto,

As others have said, don't use RMS if you want Fast I/O. Use block I/O instead to achieve the same results with block numbers (or multiples thereof) as you would with relative record number. Use the expidite flag sparingly and it could work for you.

A caveat here is that now that Rdb finally supports the reserved memory registry for global buffers, I would use Rdb. But to answer your question directly here is an example of the VMS Reserved Memory Registry, Buffer Objects and Fast I/O that I posted to the Rdb listserver back in 2002 (I think earlier but anyway).

You have two command files attached: -

rmu_init_gb.com
big_buffs.com

Good luck!

Cheers Richard Maher

PS. I use Macro as I go into Exec mode to protect memory and channels from dodgy User mode code but you don't need that. You *do* need to put your own locking in.

PPS. This example used to crash the system *VMS engineering problem not mine!* when the image exited before cleaning up. I believe this is fixed in recent versions, but see the Exec mode rundown handler for more.

[Richard J Maher]

Hi again Roberto,

The attachment on the previous post was rmu_init_gb.com, attached here is big_buffs.com.

HTH

Cheers Richard Maher

PS. Rdb refuse to use the GB vintage idea so that you could close the database yet preserve the primed VLM contents. Par for the course :-(

PPS. "I like Big Buffs that I can't de-ny. . ."

[Bob CI]

First at all, thanks to everybody for help me.

I´m collecting all your information, thanks richard for attachements, i´ll study them.

But i don´t understand a hein's note, i thought that if i can decrease the CPU overhead in

writings ( using IO_PERFORM or QIOS ), i have the CPU time available from the program´s

code and is likely that i reach the writing code more times, and then, more writing, is not

correct this ?????

[Hein van den Heuvel]

>> But i don´t understand a hein's note, i thought that if i can decrease the CPU overhead in writings ( using IO_PERFORM or QIOS ), i have the CPU time available from the program´s


That is true.
But I'm concerned with the effectiveness.
The resturn on investment.

For starterts it is not clear that CPU is a main bottleneck at all.

And secondary it is not clear that the CPU time doing (perparations for) IO operations is a significant portion of the total CPU consumed.

My suggestion is that if you can currently survive with plain RMS, possibly with sharing enabled, then there must be ample CPU.

Still, I must admit, that if you are considering writing IO functions from scratch, then you might as well use IO_PERFORM and RESERVED MEMORY and other goodies. But I wouldn't go there if I did not have a strong improvement expectation.
Maintenance, for the next engineer, could be very scary.

We only recently learned the application is using RMS relative files.
Is that all, or indexed files as well?
Is that RMS access tuned at all?
Is SHARED access being used?
Show us the typical FAB/RAB options used, and an FDL or two perhaps in a (.TXT) attachement.

Cheers,
Hein.

[Richard J Maher]

Hi Roberto,

Easiest thing to do is "see it in action" and then work back from there, so just do this: -

$mc sysman
SYSMAN> reserved_memory add MY_BUFFS -
/allocate/page_tables/zero/size=1/sysgbl
$@rmu_init_gb
$@big_buffs
$run test_buffs

Then you can see the i/o to disk and the VLM. Let me know which bits don't make sense. (As I said the User-Written System Service strategy is only there to stop stray pointers or system service calls in User--Mode code (such as TEST_BUFFS.COB) from stomping on the DB engine stuff. Not needed if you don't want it.)

This was a truly amazing bit of VMS functionality! (7.3? 7.2-2?) Was Christian Moser responsible for a lot of this - where's he now?

Don't foget that the Oracle SGA has been able to use VLM on VMS since year dot.

Cheers Richard Maher