RMS Indexed file update when Catastrophic Host Failure

[HDS]

Hello.

I understand that there may not be an absolute answer to this question and accept that there may be too many variables involved. But, the only true unanswerable question is one that goes unasked.

OpenVMS V8.3, Alpha and Itanium systems, RMS Indexed files.

When an update/write is requested to an indexed file, generally speaking, does the update occur 'first' to the data and then to the index, visa-versa, or does its order vary depending on the io subsystem hardware, disk head positioning, sector updating algorithms of io controller caches...or a mix of the above, or something else entirely? In other words, as far as VMS and RMS are concerned, would the host system give update priority to the data portion over the index? Or, could they be handled equally as both are really simply seen as data?

The underlying purpose to my asking is that a recent discussion raised the point that, although not known for sure (no HP reps in attendance), OpenVMS engineering would likely provide precendence to the data portions, for as long as data is laid down to disk, indecies can be rebuilt, files repaired, files rebuilt, etc, with reliable results in respect to business data integrity.

This topic came about as a result of inquiries to data reliability on the actual disk storage after a catastrophic failure of a host system.

This is related to plain vanilla RMS indexed file updates. No dbms, transaction processing, application io handler, etc.

Thank you in advance.

-Howard-

[Hoff]

The classic RMS strategy is (was) known as "careful writes", meaning that the writes and modifications are (were) expressly sequenced to avoid leaving corruptions in files.

If you want more than that, see the RMS journaling product.

I would strongly recommend you reconsider both your approach toward working with RMS files, and your approach to asking questions, as both appear incorrect.

Why?

You need to deal with this write-processing outside of RMS irregardless of careful writes as you will need backups of your file contents (failing to have on-line archiving is a typical design mistake), and you will also want the official answer directly from HP OpenVMS Engineering and not comments from this bozo here in ITRC.

[abrsvc]

RE:

"you will also want the official answer directly from HP OpenVMS Engineering and not comments from this bozo here in ITRC."

For an official response, I would agree. Should you want to know how things work, I'll state publicly that there are few that know RMS better.

Dan

[Hein van den Heuvel · Accepted solution]

RMS does careful write UNLESS the application requests DEFERRED WRITE. That later option is not a default, nor a file attribute, so it is always a conscious choice by the application to risk integrity for the want of speed.

In the normal careful write mode, would/will always write out date pointed to first, then the pointer. It will always write an update storage (AREA descriptor) first before writing the data allocated from it.

RMS will do its IO's SEQUENTIAL, waiting for each to finish before issueing the next. The application may choose to have that series of IOs happen asynchroneous to the application.
(RAB$V_ASY)
The storage subsystem may or might not propagate the the data to the actual drive before declaring 'done'


When deferred write is activated, RMS will only write out when...
1) a bucket when a lock request for that bucket comes in.
2) all buckets on a sys$flush call
3) and LRU bucket when a new buffer is needed and all buffers are in use/dirty.

fwiw... the INDEX is the least important.
As long as the NEXT pointers point to a valid bucket the file fiel can easily be fixed with a convert (right Bob G? :-).

1) Area (if new bucket allocated)
2) New Data bucket
3) Prior Data bucket
4) Index Bucket
5) Alternate Index structure

Cheers,
Hein.

[John Gillings]

Howard,

>This topic came about as a result of
>inquiries to data reliability on the actual
>disk storage after a catastrophic failure
>of a host system.

Rewording Hoff's answer...

The design intent of RMS is that if some catastrophic failure occurs, the integrity of the file will remain intact. The status of a particular transaction, which may have been "in flight" at the time of the failure, is either "completed" or "lost".

You don't need to go deeper and think in terms of "data" and "index". The details of how RMS achieves its oblective are irrelevant. From your perspective it's just a file.

This is an absolute answer, that's how RMS works. If you find that after a power failure or system crash, some RMS file appears to be corrupt, that is a bug, and should be reported to RMS engineering.

That said...

Please realise that this guarantee applies to a single operation on a single file. In the real world, application level transactions ("business logic") usually involve several inter-related file-level transactions. Thus, if an application executes several RMS update operations in sequence, or even in parallel, and a system failure occurs, there's no way to know which, if any, of those transactions survived, and which did not. From the RMS perspective, all files are intact, but at the business level, there may be logical corruption, as multiple files, or multiple records within one file may be in inconsistent states.

To deal with this kind of issue, you need something like RMS jornalling, or a data base product, which allow you to express business level logic like:

start transaction
operation 1
operation 2
operation 3
etc...
commit transaction

The "commit" is the event which ensures that the set of operations within the transaction are either all completed or none are completed. If a system failure occurs between "start" and "commit", your files will be in the state they were in prior to the "start" event.

[Hein van den Heuvel]

[sorry 'bout the bad/hasty spelling in the prior reply]

>> In other words, as far as VMS and RMS are concerned, would the host system give update priority to the data portion over the index? Or, could they be handled equally as both are really simply seen as data?

Maybe this warrants a detailed answer....

Only RMS knows which buffers contain data buckets, index buckets or other meta data.
Once RMS issues the QIO WRITE and lets go, nothing else knows the meaning of the contents. No special flags are passed along.

Hein

[Hein van den Heuvel]

>> Please realise that this guarantee applies to a single operation on a single file. In the real world, application level transactions ("business logic") usually involve several inter-related file-level transactions. Thus, if an application executes several RMS update operations in sequence, or even in parallel, and a system failure occurs, there's no way to know which, if any, of those transactions survived, and which did not.

For a single process there is. It will all happen in sequence... unless DFW or ASY is requested.

Following catastrophic failure, or even a simple image rundown/process termination you of course do not know how far you got, but the step will have happened in order.

RMS Run Unit journalling (RUJ - a severly undervalued RMS option free for enterprise users) can ensure that either all or nothing of the transaction becomes visible and becomes visible in 'one shot'. Protection against failure is a minor feature IMHO as it is only rarely in play (but it matters when it happens!) Protection against concurrent access is an all day, every day, necessity, again IMHO.

Cheers,
Hein

[Hoff]

SCSI and Fibre Channel gear has all sorts of weird failure modes possible, depending on what was in cache where when the mains dropped. The VMS implementation effort around careful writes, for instance, can be defenestrated when the controller re-orders I/Os and the power or the controller firmware goes bye-bye at the right (wrong) moment.

Or you could port and use CouchDB or something else, well, different than RMS. RMS is (still) nice, but tends to be a little clunky, all things considered.

And again, ITRC is not the best spot for HA and DT designs.

[Robert Gezelter]

Hein,

Yes, it is fun when the bucket infrastructure gets corrupted.

The basic RMS design is good, although there are presumptions from earlier days. I for one am inclined to include a "whole record integrity check" (e.g., CRC) to ensure that the intervening blocks in the bucket are always intact.

- Bob Gezelter, http://www.rlgsc.com

[HDS]

Hello.

Thank you all for your responses. Excellent points, excellent suggestions. Confirmation to what I was believing to be the case, along with additional items that I had not known.

I do understand and abide by the practice of using application data backups as a best-practice. No question.

In a case like this, I do agree that HP may be the best source for information, but I do enjoy and learn quite a bit from the responses that I get on the forums from such learned professionals. The views on this forum have always proven to be informative and valuable.

Thank you Hein for reminding me about the handling of deferred and async writes. In this case, I was looking at the scenario in the simplest of lights, but yes they too should be considered as well.

As to transaction processing, yup ...understand all of what has been said and agree. For this case, I was looking to see this from a simple single file perspective, albeit there is an absolute importance to the way that the application updates its business data (ex: file1 then file2 then file3...etc).

I did get the info I needed. Thank you all.

-H-