Determining when records/data were updated or inserted in RMS Idexed Sequential Files

[Vivian Taylor]

I am involved in a project which is trying to move data from a VMS system. The data is stored in RMS Index Sequential Files.

We will initially move all the data from the VMS system via FTP. Thereafter we intend to move only the updated or inserted records/data from the RMS Index Sequential Files.
The problem is there is no date/time field in our files and thus we have no way of determining when records were updated or inserted.

Q) I would like to know of any way of determining which records were updated or inserted the previous day as we don’t have a date/time field in our files? I would also like to know if RMS Index Sequential Files have a low-level time stamp in the record structure?

[Wim Van den Wyngaert]

Vivian,

No such thing in the records.

You will have to compare the file with a backup of the day before.

Wim

[Peter Barkas]

You could consider the use of RMS Journalling.

Alternatively, if you have access to the source code you could 'grow your own' by amending the update/put/delete calls to also write a transaction log of changes.

[John Gillings]

Vivian,

Write yourself a little program to do an the comparisons against a file containing the records you've already sent. It's a variant of the classical "sequential master file update" algorithm.

Simple stuff...

Read through the "master" file and the "live file" in parallel comparing records. Any in the live file that are different or missing from the master, transmit to the remote system and update in the master file. Do records get deleted? What if there's one in the master file that isn't in the live file?

[Antoniov.]

Vivian,
welcome to vms forum.

I think RMS journaling cna help you.

Antonio Vigliotti

[comarow]

You could add a short field

[Hein van den Heuvel]

The optional RMS After-Image and Before Image Journaling options can be used to gather an overview of which buckets where changed, but it do not drill down to the record level, and there is no supported tool to analyze the journal. It is only intended to be bring a backup into a know state.

The RMS buckets and records in those buckets contain no time stamp. There is a bucket check byte which is incremented on every change, but it is only a byte so it could give a false no-change every 256 updates. It could serve as a very speedy way to 'glance' over the file for infrequent changes, but again there is not standad tool for that.

New records get a fresh RFA which is garantueed to be unique and continuesly increasing (within a bucket). It has the bucket VBN plus a 'record ID' maintained in that bucket. This would allow you to store a 'high water mark' per bucket, but that only catches inserts, not updates, nor deletes.

In the end I suspect you are stuck with a daily/weekly comparision run against a known backup. The standard DIFFERENCE tool could readily be used for this purpose, notably with the /SLP output. Still, I'd opt for a home grown read & compare tool.

For the purpose of speeding up the compare I would convert the file with a larger bucket size than usual, perhaps even set to the max (63). The compare is best done with read-only access allowing only readers such that all locks are avoided. At least you should use 'NQL' no-query-lock to avoid record locks. Also... a home grown progam could readily process the file in parallel in multiple key-range chunks.

hth,
Hein.

[Jan van den Ende]

Vivian,

it all boils down to: How high is your need?

The ultimate solution would be to modify the application, and add a record item with the timestamp upon each write.

Whether that much effort is warranted is a totally site-specific decision...

Sorry, no better help than that.

Proost.

Have one on me.

jpe

[John Gillings]

Take home lesson...

Application code should always abstract all data stores (including files) into a procedural interface that defines the operations on the store in terms that are convenient for the application. Details like file accesses and physical record layouts should be hidden and irrelevant to the application code.

Easier to write the code, and MUCH easier to maintain and modify.

If that had been done in this case, the problem at hand could be solved easily by modifying the implementations of the Insert, Delete and Update operations. There are many ways it could be done - add timestamps to the record, log the any changes to another file, or even sent the updates directly to the remote system. You could even modify the whole module to operate directly on the remote data store, using any convenient transport or protocol, instead of an RMS file. Since it's all done in a layer below the application, it's transparent.