topic Re: RMS RFA GET Changes next record pointer? in Operating System - OpenVMS

RMS RFA GET Changes next record pointer?

Patricia Morman — Mon, 27 Sep 2010 07:56:20 GMT

I have a major application written (by me) in VMS BASIC that has been running for thirty-some years. It is being replaced and I am rewriting one of the modules (a warehouse module) to interact with the new ERP system. The RMS IO is all done by one function written in the 80's. Solid as a rock, right? Well, I came across what I thought was just a simple program bug today and it has turned out to be something major. I have had the assumption that doing an RFA GET on a record does not change the next record pointer, whereas apparently it does. If I am reading a file and I hit a locked record, the IO function waits for a second and then tries again, this time by RFA, and keeps trying until it has success. But if this happens when I'm reading in an alternate key sequence, I'm thrown back into primary key sequence, which makes me think that the RFA read must be changing the next record pointer. I was simply amazed - how could I have never noticed this before? But in the established application I had "key files" for all of the major files and accessed the data file itself by RFA only when using an alternate sequence, so I guess it's possible that it just never came up. But in this new application I have eliminated the key files altogether. The systems are so fast now, it really isn't necessary. So I have a problem. There are two scenarios. 1. I'm reading in an alternate sequence and come across a locked record. I can probably just redo the same get(?). But 2. I'm going through the file in an alternate sequence and I'm not locking records, but I come across something interesting and want to change something on the record. So I get the record by RFA to lock it, I update the record and continue. But this doesn't work because the RFA get changes my sequence. I can't use position to reset myself on the alternate sequence because I might have duplicate keys. So how do I set myself straight again? The only thing I can think to do is to have the file open on another channel and when I want to lock a record to make changes, get it by RFA on this second channel, keeping my sequence intact on the first channel. But there's got to be a better way, right?

Re: RMS RFA GET Changes next record pointer?

Hoff — Mon, 27 Sep 2010 11:16:33 GMT

Quite the wall of text.

Paragraphs help with intelligibility.

I'd expect to see the pointer change here.

Open and use a second record stream for this task, as you'd considered. One stream runs the sequence, and the second stream runs the updates. Running multiple streams is easy, and it avoids the collisions between unrelated operations, and it avoids changes to the sequencing.

I'm heavily changing these file-access designs of late, including hauling whole blocks of data into 64-bit virtual memory (64-bit addressing is not an option for BASIC, AFAIK), or moving from RMS into a different database. RMS is a reasonable and high functional NoSQL, but there can be other choices.

Hauling some older file-based designs into, for instance, data arrays, has produced substantial application performance improvements. And it made the code simpler. Profile your code via DECset, and see where you're spending the wall-clock. I/O tends to be one of those time-consuming areas, too.

And when still working with RMS (and not a database that has archival capabilities), consider implementing a mechanism to get off of a current record, either on request, or a timeout. This to unlock any current record. Or avoid locking records. This analogous to that lock-retry sequence, but for those streams that already have the record, and are sitting on it. This all gets tied into parallel activities and particularly with getting a reliable and consistent backup of the file(s) involve. When you're working with RMS, your application code is central to getting on-line backups, if you want that capability.

Re: RMS RFA GET Changes next record pointer?

Hein van den Heuvel — Mon, 27 Sep 2010 11:29:56 GMT

>> I have had the assumption that doing an RFA GET on a record does not change the next record pointer

I believe that's only tru for FIND, not GET.
Table 8.3 "Record Access Stream Context" in the Guide to File Application lists the various effects.

http://h71000.www7.hp.com/doc/731final/4506/4506pro_024.html#apps_recrd_acces_stream_context

>> If I am reading a file and I hit a locked record, the IO function waits for a second and then tries again

Yuck. The ringer-less phone algorithm. (Pick up the phone to find out if someone is calling, instead of waiting for the phone to ring when someone called).
1) In high contention environments you'll loose you place in the list which can lead to lock-outs in extreme cases
2) The system does not know some process is more or less waiting for the record so it can not help detects issues (deadlock detection, getlki blocked-by, anal/system show proc/lock, DecAMDS)
3) The wait is always 1 second, even if the record becomes available in a milli second.

Why not use the RMS WAT + TMO option which is available in BASIC with the WAIT lock clause? (answer: you may want to tell the end user early about a potential wait).

>> There are two scenarios. 1. I'm reading in an alternate sequence and come across a locked record. I can probably just redo the same get(?).

Yes... with a caveat for file with duplicates on the alternate keys. A keyed get will get you back to the first in the (potential) series of duplicates. I have at time needed to read forward to re-find the RFA for the duplicate where I left off.

>>> But 2... So I get the record by RFA to lock it,

Use FIND ?

>>> can't use position to reset myself on the alternate sequence because I might have duplicate keys

Good catch!

>>> The only thing I can think to do is to have the file open on another channel and when I want to lock a record to make changes, get it by RFA on this second channel, keeping my sequence intact on the first channel.

No, that is actually a good, reliable and very efficient solution providing the second channel shares the same open using a CONNECT + MSE, accessible in BASIC through OPEN ... CONNECT chnl-exp2 ...

The target buffer(s) will surely be there in the buffer from the main channel. Just data and potential RRV bucket access. No more and no less work than using the original channel.
Maps well onto SQL like re-implementations.

If more understanding is needed then you may want to try a test program in native RMS (called from C, Macro or Basic as you please) before trying BASIC to make sure the BASRTL is not trying to help too much.

Hope this helps some,
Hein RMS van den Heuvel

Re: RMS RFA GET Changes next record pointer?

Hein van den Heuvel — Mon, 27 Sep 2010 12:59:14 GMT

>> If I am reading a file and I hit a locked record, the IO function waits for a second and then tries again, this time by RFA

I forgot to ask... How does the program even get the RFA, unless you did a GET REGARDLESS ??

RMS does NOT update RFA nor record buffer on failure.

Attached a self-contained C program to create an 2 key indexed file put some records read and lock on one rab, read by (descending) alternate key on an other stream and get a locked error. (yes, the program is sloppy-printing record buffer contents)

Sample output below.

Should be easy enough to modify to your experimentation needs, flipping to $FIND and/or to use: rab$b_rac = RAB$C_RFA

Cheers,
Hein

$ run NEXT_RECORD
PUT record (4,1) 0001 blah
PUT record (4,2) 0002 blah
PUT record (4,3) 0003 blah
PUT record (4,4) 0004 blah
LCK record (4,1) 0001 blah
LCK record (4,2) 0002 blah
GET record (4,4) 65537 0004 blah
GET record (4,3) 65537 0003 blah
GET record (0,0) 98986 0003 blah
Closed = Unlocked = 65537. Retry.
TRY record (4,2) 65537 0002 blah

Re: RMS RFA GET Changes next record pointer?

Hein van den Heuvel — Mon, 27 Sep 2010 13:03:43 GMT

Attach failed.

Or rather... the reply failed and I missed the attach on the retry.

We had such a good streak of problem-less replies, but today I had to stare at a blank screen twice instead of the usual 'ack'.
I guess we'll have to protect ourselves again with an extra save of the text-to-be-replied. Even though a browser-back often gets it back.

Anyway... re-replying with attachment as well as included source.

Hein

#include
#include
#include
#define MAXRECSIZ 50
int SYS$CREATE(), SYS$OPEN(), SYS$CLOSE(), SYS$CONNECT(), SYS$GET(), SYS$PUT(), SYS$FIND();
main (int argc, char *argv[])
{
struct FAB fab, read_fab;
struct RAB rab, read_rab;
struct XABKEY key, alt;

char recbuf[MAXRECSIZ+1];
int i, s, stat;
fab = cc$rms_fab;
rab = cc$rms_rab;
key = alt = cc$rms_xabkey;

fab.fab$b_org = FAB$C_IDX;
fab.fab$b_shr = FAB$M_UPD;
fab.fab$b_fac = FAB$M_GET | FAB$M_UPD;
fab.fab$l_fna = "tmp.idx";
fab.fab$b_fns = strlen( fab.fab$l_fna );
read_fab = fab;

fab.fab$l_xab = &key;
key.xab$b_siz0 = 4;
key.xab$l_nxt = &alt;
alt.xab$b_siz0 = 4;
alt.xab$b_ref = 1;
alt.xab$b_dtp = XAB$C_DSTG;

stat = SYS$CREATE ( &fab );
if (!(stat&1)) return stat;
stat = SYS$OPEN ( &read_fab );
if (!(stat&1)) return stat;

rab.rab$l_rbf = recbuf;
rab.rab$l_ubf = recbuf;
rab.rab$w_usz = MAXRECSIZ;
read_rab = rab;

rab.rab$l_fab = &fab;
stat = SYS$CONNECT ( &rab );
if (!(stat&1)) return stat;

for ( i = 1; i < 5; i++) {
rab.rab$w_rsz = sprintf ( recbuf, "%04d blah", i);
stat = SYS$PUT ( &rab ); // initial load record
printf ("PUT record (%d,%d) %s\n", rab.rab$l_rfa0, rab.rab$w_rfa4, recbuf);
}

for ( i = 1; i < 3; i++) {
stat = SYS$GET ( &rab );
printf ("LCK record (%d,%d) %s\n", rab.rab$l_rfa0, rab.rab$w_rfa4, recbuf);
}

read_rab.rab$l_fab = &read_fab;
read_rab.rab$b_krf = 1;
stat = SYS$CONNECT ( &read_rab );
if (!(stat&1)) return stat;

while (stat & 1) {
read_rab.rab$l_rfa0 = 0;
read_rab.rab$w_rfa4 = 0;
stat = SYS$GET ( &read_rab );
printf ("GET record (%d,%d) %5d %s\n", read_rab.rab$l_rfa0, read_rab.rab$w_rfa4, stat, recbuf);
}

stat = SYS$CLOSE ( &fab );
printf ("Closed = Unlocked = %d. Retry.\n", stat);

stat = SYS$GET ( &read_rab );
printf ("TRY record (%d,%d) %5d %s\n", read_rab.rab$l_rfa0, read_rab.rab$w_rfa4, stat, recbuf);

}

Re: RMS RFA GET Changes next record pointer?

Patricia Morman — Tue, 28 Sep 2010 02:36:38 GMT

Thanks very much for your replies.

1. I will use WAIT 1% with GETs.
2. I will use OPEN CONNECT and have two streams going with the files, one stream to do the reading, another to do the random updates.

I really appreciate your generous help.

Re: RMS RFA GET Changes next record pointer?

Jan van den Ende — Tue, 28 Sep 2010 07:00:16 GMT

Patricia,

>>>
I really appreciate your generous help.
<<<

In these forums, appreciation is shown by assigning points.

Proost.

Have one on me.

jpe

Re: RMS RFA GET Changes next record pointer?

Patricia Morman — Tue, 28 Sep 2010 07:05:47 GMT

Sorry, had done it previously but guess I didn't submit them.

Re: RMS RFA GET Changes next record pointer?

Peter Barkas — Thu, 28 Oct 2010 13:18:05 GMT

1. GET with lock with WAIT 1 (say).

If record lock/timeout, can use FIND to get the RFA to find out who has it locked and broadcast or display message etc.

Repeat GET with lock.

2. GET with lock and unlock if not interested.

Simples.

Re: RMS RFA GET Changes next record pointer?

Patricia Morman — Thu, 28 Oct 2010 20:10:52 GMT

Thanks again.