io atomicity

[Scotty HD]

when doing io to a file using qio, the io can get segmented. i hope this understanding is correct. in such a case is the io guaranteed to be atomic ?

also how about atomicity when doing io to file using rms ?

Scotty

[Robert Gezelter]

Scotty,

Take a step back. What precisely do you mean by "atomic"?

Even an IO that is not "segmented" can be interrupted in progress (e.g., device faults, power conditions). The answer to your questions will depend precisely upon the precise definition of "atomic". Put another way, "Atomic with respect to what?"

The same qualification applies to RMS files. RMS locking guarantees exclusive access to the contents of the file. It does not say anything about the rest of the device.

- Bob Gezelter, http://www.rlgsc.com

[Scotty HD]

#Robert Gezelter
sorry for not providing all the data.

#Atomic with respect to what?
i meant reader io atomic with respect to writer io to file which is
accessed at the same time.

assume,
-IO gets segmented to two segment
segment_1: block x, which has data x1
segment_2: block y, which has data y1
-reader and writer are accessing the file at same time
reader: reads the data in block x and y
writer: writes data x2 to block x and y2 to block y

after io completes,
1) no atomicity and reader can get mix data
a) reader end up with data of x2 in x (new data) and y1 in y (old data)
or
b) reader end up with data of x1 in x (old data) and y2 in y (new data)
2) atomicity ensure that reader will get
a) x1 in x, y1 in y (all old data)
or
b) x2 in x, y2 in y (all new data)

what would happen in case of qio access ?
and
what would happen in case of rms access ?

Scotty

[Hein van den Heuvel]

When you say segmented, I think you refer to what OpenVMS labels SPLIT IO. This is where an virtual IO to sequence of consecutive blocks in a file are mapped to multiple non-consecutive physical block on the disks due the file fragmentation.
They are counted by MONITOR on the IO page.

An IO request will only return back to the application when is is completely done, split IO or not. If the IO was split then all underlying IOs must have completed before the calling application layer (which might be RMS) is told to continue.

With a split IO the CHANCES that an virtual IO only happens partially due to disk or full system failure increase, but remain small.
Uncoordinated readers, like BACK/IGNO=INTER will also have an increased risk of getting a partially updated chunck of data.

RMS has no magic powers over QIO.
Any rule that applies to QIO applies to RMS.

Now if you think of an RMS $PUT to an indexed file as atomic, then that change the dynamics a lot. Split-io/fragmenation becomes largely irrelevant compared to the mutliple updates RMS may have to make to data, index, area-allocation. RMS, without deferred write, will do IO in a fail safe order, making sure to update targets before pointers and update allocations before using the allocated blocks.
The timing window for things like BACK/IGNO=INTER to get inconsistent data is enormous though... as long as it takes to do the backup.
If it very well possible that backup picks up stale pointers to data make some data inaccesible (notabley when bucket splits are happening) and for sure it will have stale allocation indicators as the allocation details are stored in the first blocks of the file.

If you'd like to know more, then please explain a little better why the question is being asked. Some design option? Observed failure? Failure protection (what kind of failure?)
Transactional protection? (RMS RU journalling might help)

Hope this help some,
Hein

[Hein van den Heuvel]

Hmmm, new details arrived while I was replying.

One missing details is perhaps whether the reader and writer at least issue the IO for the same single chunk of blocks. 'a buffer'.

Real ,physical disks, which might do IO re-ordering (elevator algorithms) notably when busy causing any and all combination because x might physically be before or after y.

But these split IOs are necessarily file IOs.
So other factors play a role. For example typically the XFC will first have caught the whole write. In that case you'll see all old, or all new depending on who knocked first.

With RMS, and sharing, this is certainly the case because the writer will take out a lock on the whole buffer (my opening question) and the reader will only start reading when that write lock is released after the IO completes, split or not.


Hein

[Hoff]

Neither segmented transfers nor non-segmented multi-block transfers are typically atomic.

Discussions of atomic operations require understanding the hardware and caching and the host OS and I/O path in some detail, as well as what are typically hierarchical storage requirements within these applications.

Within classic disk storage, the only thing in this neighborhood that might be atomic is a single sector write. That usually gets to the disk media, or not. Note the use of the word "usually" there, too. Multiple-block transfers may or may not get written to the storage media in their entirety.

A multi-block I/O transfer (whether segmented or unsegmented) typically has no guarantee of atomicity.

Should you be considering doing synchronous writes for reliability, you're using a simplistic design from around 1990 or so, and the performance of these designs will be poor. If you're not careful, you'll be limited by the speed of synchronous writes across multiple spindles. (Which in this era is glacial.)

You'll also be looking at so-called careful writes, a feature which is built into RMS. This means that a sequence of I/O writes are either added to the file or not, but the file design remains consistent.

As for understanding the I/O path, VMS itself largely lacks write caching, though the storage controller(s) and the disks involved generally do include caches, and (when the power fails) the contents of these caches can be lost. If there's no less-interruptable power supply (LIPS) or batteries involved, assuming they're working.

Some RAID controllers have batteries for the cache.

The usual design approach with persistent storage is to look at the particular problems you are seeking protecting against, and whether (for instance) multiple hosts and replication might be appropriate (this can be leagues faster than hard disks), usually with more modern (and faster) storage.

Hard disks are increasingly being relegated to use as archival storage, too; as a replacement or an intermediate tier on the way to tapes.

You'll also end up looking at recovery processing in some detail, such as the failures that can arise during RAID-5 recovery; some RAID levels are particularly prone to triggering secondary and catastrophic spindle failures.

Depending on what you're up to, a transactional solution can be appropriate. Or a log file. Or replication.

Most folks will look at using a database here, and allowing the database designers to deal with these sorts of questions and quandaries. A database certainly isn't a panacea, but it does leverage existing code and existing experience with the problems that arise within the I/O stack. Rolling your own gets expensive.

Your question unfortunately leaves rather much assumed and more yet unmentioned, as others here have alluded to. This around your version(s), application requirements, proposed designs, and other such. Point questions of specific features often won't get you the best answer, and may well get you what amounts to the wrong answer. Upping your game with your questions will help your career, too.

[John Gillings]

Scotty,

Just to add confusion to the mix, I'm going to question your use of the term "segmented". I'm not aware of any standard meaning for this term in $QIO or RMS.

[aside, are you really doing file I/O with QIO? Why make life hard for yourself? Use RMS instead]

One place where there IS an object called a "Segmented Record" is in FORTRAN. Here's the definition:

"A segmented record is a single logical record consisting of one or more variable-length, unformatted records in a sequential file. Each variable-length record constitutes a segment. The length of a segmented record is arbitrary.

Segmented records are useful when you want to write exceptionally long records but cannot or do not wish to define one long variable-length record. When writing unformatted data to a sequential file using sequential access, the default record type is segmented."

Since each segment is effectively an independent record from the RMS perspective, if you were to WRITE a segmented record from a FORTRAN program, whilst READING the same file from a non-FORTRAN program using standard RMS calls, you may well encounter timing issues where the non-FORTRAN program sees partial records. Segmentation is a layer of structure imposed by FORTRAN over RMS. If you have files with segmented records, you'd be well advised to only access them from FORTRAN, rather than have to write your own code to pack and unpack the logical records from constituent segments.

That said, I don't think you're taking about FORTRAN.

From an RMS perspecive, a "record" is atomic. When you write a record, it's either written to the file, or, )if a system crashes or suffers a power fail during the operation), it doesn't make it. There's no such thing as a partial record.

[GuentherF]

And how is this related to the same question you asked in the POSIX forum a while ago?

/Guenther