SET FILE/NOBI_JOURNAL

[HDS]

Hello.

We have a file configured with Before-Image RMS Journaling. It was configured using
$SET FILE/BI_JOURNAL=(FILE=,CREATE)

Immediately prior to the enabling of BI journaling, we use a method of sw locking to synchronize accesses and guarantee that everyone if out of the file; no accesses at all.

Once the BI journaling is enabled, again using a method of sw synchronization techniques, we prohibit user updating of the target file and allow users to only query the file (Open Shared-Read, record RRL/NLK).

At that point, we run a batch process that selectively updates specific records in that file, synchronizing those record updates with any of the possible concurrent reads against a target record. (It is actually that batch process that performs the enabling of the BI journaling.) The BI journaling to that target file is being done to allow a quick recovery of that file if a problem were to occur in the batch process. This file is very [very] large; a $BACKUP of it would take too long. Note that we would disallow any accesses to that file if a recovery is to be performed.

Once that batch updating is completed, we are looking to disable the BI journaling (no longer required for recovery methods for that batch process) re-permit the users to have update access to the file.

We can control the sw locking/synchronization to the file, but cannot $SET FILE/NOBI disable the journaling when users have query accesses being performed.
%SET-E-READERR, error reading
-SYSTEM-W-ACCONFLICT, file access conflict

We understand reasons for $SET FILE requiring an EX lock in this case, but we are aware and do control the accesses to the file via sw techniques.

So, we are looking for a means [if any] for disabling BI journaling from a file (maybe using a SYS$OPEN and an update to the XABJNL fields) while a given file is OPEN SHARED-READ and records are being accessed via RRL/NLKs.

Note that we have been discussing [internally] the possible necessity to temporarily [and quickly] remove all accesses, perform the $SET FILE and then re-allow accesses....but this is, in short, a business inconvenience that we'd like to avoid.

We genrally use Fortran/77 and Fortran/90, and 'C'. Any assistance is appreciated.

Thank you.

[Hein van den Heuvel · Accepted solution]

What OpenVMS version? Just curious. It will not matter much as SET FILE/SHARE is only checked for /[no]GLOBAL and /[no]STATISTICS.
It will flip fib$m_nolock before the QIO IO$_ACCESS in set file for those two options only, if specified.

Clearly doing a BI recovery can not be done with any shared access as no bucket locks are used. So a reader could have a bucket's after-image in its buffers, and the BI recorvery could easily invalidate the targets of pointers in the (read only, but stale) buffer.

To walk away from BI journalling while users are reading, and even while writing, sounds like conceptionally possible in a careful controlled environment as described. It would be too much to handle as a general feature with too many options to shoot oneself in the foot.

Whether a file is RMS journalled or not, is flagged through an ACE in its file header.
$DUMP/HEADER and $ANALYZE/RMS (and the OpenVMS sources!) can provide insight for the details.

The set file /journal code uses QIO Open and eventually uses a QIO IO$_MODIFY with ATR$C_DELACLENT. I suppose one could conceivably call this after opening with fib$m_nolock.

But in your case, why bother? It sounds like the application is very close to be able to handle this with the official tools.
Apparenyl there is already the knowledge to close and re-open the file triggered by sharing situation. All you need is an extra interlock to wait between close and re-open for some gloabl flag/lock to indicate the attributes have been changes. Some 'wait-for-reopen flag in the lvb used to trigger the close. That is assuming blocking ast's are used to implement application level synchronization.

Hope this helps a little,

Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[HDS]

Good morning and thank you.
That response does offer insight into this issue.

I will present this info to my team members and will get back to you.

Much obliged.

[Wim Van den Wyngaert]

May be a stupid question but why don't you leave BI active all the time ?

Wim

[Hein van den Heuvel]

[There are no stupid questions!]

Probably because it would generate too much data under normal production changes where rollback is less obvious anyway.

For batch job it is relatively clear when it started, and possible what else changed.

To be able to back out the changes done just during a particular backup may be limited/controlled in volume, and desirable to be able to undo.
Indeed this may be a lot faster than restoring a backup if/when a job fails in the middle.

BI journalling writes a log of each and every used byte for every bucket modified. Change a bit in a record in a bucekt with 20 records and the whole old bucket contents (up to 'next free byte') will be appended as a timestamped variable length record to a change log file.

Cheers,
Hein.

[Wim Van den Wyngaert]

Recovery enabled during 1 batch job and not during other activities ? Seems a strange strategy to me. A crash can happen any time and recobery will only be possible during this batch job.

Wim

[HDS]

Greetings.

I am still discussing this with my companions. You have provided much for us to discuss. Many thanks.

As to the "why BI just during that batch job?....what about other times?" You are correct; that is a reasonable point. I should note that we have other means (such as full file backups) that cover us for other times of the day. In this particular case, we know that no one but our single batch process is performing updates to that specific file during that time period. We wanted a fast way to recover that very large file if a failure should occur just during the execution of that process.

As to "Why not just leave BI on always?"... That has been considered and may possibly be an outcome. We'd prefer that not to be the case because of the performance hit and the data volume in the journal, in addition to the time periods being specified on the recovery. Note that the idea may be chosen "partially", as we may be able to leave the BI on until some later point where we know that the data file is once again in a "not available" state where no one is querying (such as during nightly updates, etc...cases that use full file backups and hw replication for recoveries).

Again, many thanks to all.

[HDS]

Hello.

I wish to thank all of you. As it happens, we are going to hit this from the application, enabling and disabling BI journaling at points where we know no users are in the file. It seemed to be the most straight-forward approach.

Oh...by the by...it is OpenVMS V8.3 on ALPHA systems. (I had not answered that one question.)

Thanks again...