Re-read a block written by another process

[Ben Armstrong]

When I re-read a block written to a fixed-length record (512 bytes) file by another process using read() in C, I get an old block from the file, not the new block that was just written. What am I doing wrong?

Sample test run using attached test program:

$ cc test
$ link test
$ test=="$sys$disk:[]test"
$ test write 2 ! Fill the first block with chr(2)
$ test test ! Fill 1st block with chr(0) -> chr(9) and show first byte
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2
First byte: 2

Thanks,
Ben

[Steven Schweda]

Step one: Look at the status from
lib$spawn():

alp $ gdiff test.c_orig test.c
19a20,23
> printf( " spawn sts = %%x%08x .\n", libspc_return_status);
> if ((libspc_return_status& 7) != 1)
> printf( "%s\n", strerror( 65535, libspc_return_status));
>

alp $ test test
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2
spawn sts = %x00158224 .
invalid string descriptor
First byte: 2

Then, try using a string descriptor for the
command.

[David Jones_21]

Ignoring the broken reproducer program, I feel your pain Ben. The only way I could ever get something like that work was to freopen the file every time.

I think you'll save yourself time and frustration by ditching the CRTL and working directly with RMS calls (documentation is better at least).

[Steven Schweda]

With some better diagnostics, you can even
see what goes wrong then:

alp $ gdiff test.c_orig test.c
6a7,8
> #include
> #include
12a15
> $DESCRIPTOR( buffer_dsc, buffer);
16c19
< libspc_return_status = lib$spawn(&buffer,0,0,0,0,0,
---
> libspc_return_status = lib$spawn(&buffer_dsc,0,0,0,0,0,
19a23,26
> fprintf( stderr, " spawn sts = %%x%08x .\n", libspc_return_status);
> if ((libspc_return_status& 7) != 1)
> fprintf( stderr, "%s\n", strerror( 65535, libspc_return_status));
>
37c44,45
< fprintf(stderr,"Error: open failed.\n");
---
> fprintf(stderr,"Error: open (r+) failed.\n");
> fprintf( stderr, "%s\n", strerror( errno));
58c66,67
< fprintf(stderr, "ERROR: open failed\n");
---
> fprintf(stderr, "ERROR: open (w+) failed\n");
> fprintf( stderr, "%s\n", strerror( errno));


alp $ test test
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0
ERROR: open (w+) failed
target record currently locked by another stream
spawn sts = %x00000001 .
First byte: 0

But at least the "First byte" isn't "2" now.

[David Jones_21 · Accepted solution]

I attached a program that demonstrates the problem intended by Mr. Armstrong. Replacing the fseek with an freopen will make it produce the output originally expected.

[Ben Armstrong]

Meh. Sorry about the broken test. That was cobbled together quickly as a port from a ruby script that reproduced the problem correctly. But I figured nobody here uses my VMS ruby port (http://vmsruby.rubyforge.org) so a C version was more appropriate.

[Ben Armstrong]

Thanks for the fixed test program, David.

So if I'm going to resort to direct RMS calls, can you point me in the right direction? I've been lost in the doc for a couple of days on this problem, mostly focusing on the CRTL, but studying some of RMS too, and it's ... well ... just a bit overwhelming.

What do other porting projects do? VMSPerl perhaps? Or are my needs atypical?

Ben

[Ben Armstrong]

Actually, looking at your fixed test, it doesn't appear to demonstrates the problem I am having. It shows that the writes fail. In my ruby example, the writes succeeded. That is, opening the file "shr=get,put,upd" allowed it to be opened by the other process and written, which I independently verified by interrupting the test half-way through and performing a dump on the file after the spawned write ... the file did contain chr(5) throughout the first block.

Changing the open mode from 'r+' to 'w+' does not correctly model my application's behaviour. The second process should not truncate the test file. It should merely open it, seek to a block within the file and update that block with new values. The first process should then be able to read (without reopening the file) the new values.

Ben

[Hein van den Heuvel]

RMS has it's rules about record locking.
I haven't studied your program in detail but I suspect you were caught by automatic record locking. When a record is read, in a shared context it is locked untill the next record is read or untill it is explcitly unlocked.

So in RMS terms you possibly want your reader to use RAB$V_RNL (+ RAB$V_WAT) or maybe RAB$V_NQL. No Query Lock roughly equals RNL + RRL, but is better as ot does not take out a (record) lock.

Normally one uses an $UPDATE call to update date records (duh!), after making the record current and locked through a $GET or $FIND. You can use $PUT (as per example) but it is not so clean and may need an RAB$V_UIF. And you'd have to decide between
sequential writing 'the next record' or by key using an 'rrn' Relative Record Number for a fixed length record file.

RMS also offers SYS$READ on SYS$WRITE which at first glance map closer to your needs. Those calls are a thin but handy wrapper around SYS$QIOW and allow simple, unsynchronized, concurrent access to file.

Hope this help some.
Maybe more later, my work first!

Hein.

[Ben Armstrong]

So there's no ambiguity about what I'm trying to test, I have attached a working version of my test.c.

My test run:

$ cc test
$ link test
$ test == "$sys$disk:[]test"
$ test create
Write value: 0
$ test test
Write value: 0
Spawn status: 1
First byte: 0
Write value: 1
Spawn status: 1
First byte: 0
Write value: 2
Spawn status: 1
First byte: 0
Write value: 3
Spawn status: 1
First byte: 0
Write value: 4
Spawn status: 1
First byte: 0
Write value: 5
Spawn status: 1
First byte: 0
Write value: 6
Spawn status: 1
First byte: 0
Write value: 7
Spawn status: 1
First byte: 0
Write value: 8
Spawn status: 1
First byte: 0
Write value: 9
Spawn status: 1
First byte: 0
$ test test
Write value: 0
Spawn status: 1
First byte: 9
Write value: 1
Spawn status: 1
First byte: 9
Write value: 2
Spawn status: 1
First byte: 9
Write value: 3
Spawn status: 1
First byte: 9
Write value: 4
Spawn status: 1
First byte: 9
Write value: 5
Spawn status: 1
First byte: 9
Write value: 6
Spawn status: 1
First byte: 9
Write value: 7
Spawn status: 1
First byte: 9
Write value: 8
Spawn status: 1
First byte: 9
Write value: 9
Spawn status: 1
First byte: 9

As you can see, only the very first value to be read in 'test' mode will be returned, even after subsequent values have been written to the test file.

[Ben Armstrong]

Oops. Forgot to include the updated test.c. Here it is.

[David Jones_21]

Here's some code using direct RMS calls to play with, of course there's a lot of refinement that can be done to it. I first tried block I/O, but it's less friendly to extending the file and managing the EOF position than $GET and $PUT.

[Hein van den Heuvel]

The biggest problem with the code written is the lack of "ctx=rec".

This causes the C RTL to request SHR=UPI from RMS meaning no interlock will take place.

To verify this use
ANAL/SYSTEM... SET PROC 'test'... SHOW PROC/RMS=(FAB,RAB)

You may want to also issue SET FILE/STAT
and SDA> SHOW PROC/RMS=FSB

Of course once you do enable full sharing, you probably have to seek to your write target and use 'rop=uif' to switch to update from writes and add something to unlock the records after the reads.

Finally... you don't need spawn to test this. Just open the file again with a different handle and the processing will be exactly similar to the multy process case, just easier to see and debug.

Verify with SHOW DEV/FILE

Good luck,
Hein.

[Hein van den Heuvel]

I changed the last program attached to use 'ctx=rec' everywhere; 'rop=uif' on the open for update and added an extra read to unlock the target after the first read for the data.

While it works, I'm still unhappy not to be able to explain several things, mostly C-rtl choices it seems.

Specifically, the RAB for outf contained an RMS-R-RLK (000182aa) status. Using system service logging i found there is an unexpected (for me) $get right after the $update. See trace below.

I could not readily find a simple 'unlock/free' call, other than asking the C-rtl for the rab address.

Also, in "C" mode with ctx=stm, I could not find a simple way to force a re-read from disk other than a crude freopen. I hope I overlooked something.

I'd say go native RMS if you can and take control!

SSL Trace below, as well as perl to pretty print the trace for this purpose.

Hein.

$ test:=="$sys$login:x "
$ set proc/ssl=(state=on,flag=arg)
$ test test
Write value: 1
First byte: 1
Write value: 2
First byte: 2
Write value: 3
First byte: 3
Write value: 4
First byte: 4
$ set proc/ssl=(state=unload)
$ set proc/ssl=stat=unload
$ anal/ssl/normal/select=acce=U/out=n.x
$ perl -ne "if (/^(\S+)\s+sts: (\S+).*!(\S+)$/){$n=$1;$s=$2;$t=$3;$_=<>;$_=<>;m/(.{8}) 2:/;printf qq(%s %s %s %s\n),$t,$s,$1,$n}" n
.x

22:57:20.42 00018001 00054da0 SYS$RMS_OPEN
22:57:20.42 000001bc 7ace7548 SYS$$TRNLNM
22:57:20.42 00000001 00000080 SYS$GETJPI
22:57:20.42 00000001 00000080 SYS$GETSYI
22:57:20.42 00010001 00057680 SYS$RMS_CONNECT
22:57:20.42 00018001 000604a0 SYS$RMS_OPEN
22:57:20.42 00010001 00061a60 SYS$RMS_CONNECT
22:57:20.42 000001bc 7ace72b8 SYS$$TRNLNM
22:57:20.42 00000001 7ace7278 SYS$EXPREG_64
22:57:20.42 00010001 00065860 SYS$RMS_PARSE
22:57:20.42 00018574 00065860 SYS$RMS_SEARCH
22:57:20.42 00010001 00067040 SYS$RMS_OPEN
22:57:20.42 00010001 00067040 SYS$RMS_CLOSE
22:57:20.42 00010001 00067040 SYS$RMS_CREATE
22:57:20.42 00000001 00000080 SYS$GETSYI
22:57:20.42 00010001 00069920 SYS$RMS_CONNECT
22:57:20.43 00000001 00000080 SYS$GETDVI
22:57:20.43 000001bc 7ace6e10 SYS$$TRNLNM
22:57:20.43 00010001 00069920 SYS$RMS_WAIT
22:57:20.43 00010001 00067040 SYS$RMS_CLOSE
22:57:20.43 00010001 00064080 SYS$RMS_CREATE
22:57:20.43 00010001 00064080 SYS$RMS_CLOSE
22:57:20.43 00010001 00064080 SYS$RMS_CREATE
22:57:20.43 00000001 00000080 SYS$GETSYI
22:57:20.43 00010001 00065640 SYS$RMS_CONNECT
22:57:20.43 000001bc 7ace72b8 SYS$$TRNLNM
22:57:20.43 00018001 00065640 SYS$RMS_PUT
22:57:20.43 00000009 00000080 SYS$SYNCH_INT
22:57:20.43 00010001 00061a60 SYS$RMS_GET
22:57:20.43 000001bc 7ace7278 SYS$$TRNLNM
22:57:20.43 00018001 00061a60 SYS$RMS_UPDATE
22:57:20.43 -------- 00000080 SYS$SYNCH_INT
22:57:20.43 -------- 00000080 SYS$SYNCH_INT
22:57:20.44 00010001 00061a60 SYS$RMS_GET
22:57:20.44 00010001 00061a60 SYS$RMS_WAIT
22:57:20.44 00018001 000604a0 SYS$RMS_CLOSE
22:57:20.44 -------- 00000080 SYS$SYNCH_INT
22:57:20.46 00010001 00057680 SYS$RMS_DISPLAY
22:57:20.46 00010001 00057680 SYS$RMS_GET
22:57:20.46 00018001 00065640 SYS$RMS_PUT
22:57:20.46 00010001 00057680 SYS$RMS_GET
22:57:20.46 00018001 00061a60 SYS$RMS_OPEN
22:57:20.46 00010001 000604a0 SYS$RMS_CONNECT
22:57:20.46 00018001 00065640 SYS$RMS_PUT
22:57:20.46 00000009 00000080 SYS$SYNCH_INT
22:57:20.46 00010001 000604a0 SYS$RMS_GET
22:57:20.46 00018001 000604a0 SYS$RMS_UPDATE
22:57:20.46 -------- 00000080 SYS$SYNCH_INT
22:57:20.47 000182aa 000604a0 SYS$RMS_GET
22:57:20.47 000182aa 000604a0 SYS$RMS_WAIT
22:57:20.47 00018001 00061a60 SYS$RMS_CLOSE
22:57:20.47 -------- 00000080 SYS$SYNCH_INT
22:57:20.49 00010001 00057680 SYS$RMS_GET
22:57:20.49 00018001 00065640 SYS$RMS_PUT
22:57:20.49 00010001 00057680 SYS$RMS_GET
22:57:20.49 00018001 000604a0 SYS$RMS_OPEN
22:57:20.49 00010001 00061a60 SYS$RMS_CONNECT
22:57:20.49 00018001 00065640 SYS$RMS_PUT
22:57:20.49 00010001 00061a60 SYS$RMS_GET
22:57:20.49 00018001 00061a60 SYS$RMS_UPDATE
22:57:20.49 -------- 00000080 SYS$SYNCH_INT
22:57:20.50 000182aa 00061a60 SYS$RMS_GET
22:57:20.50 000182aa 00061a60 SYS$RMS_WAIT
22:57:20.50 00018001 000604a0 SYS$RMS_CLOSE
22:57:20.50 -------- 00000080 SYS$SYNCH_INT
22:57:20.52 00010001 00057680 SYS$RMS_GET
22:57:20.52 00018001 00065640 SYS$RMS_PUT
22:57:20.52 00010001 00057680 SYS$RMS_GET
22:57:20.52 -------- 00061a60 SYS$RMS_OPEN
22:57:20.52 00010001 000604a0 SYS$RMS_CONNECT
22:57:20.52 00018001 00065640 SYS$RMS_PUT
22:57:20.52 00000009 00000080 SYS$SYNCH_INT
22:57:20.52 00010001 000604a0 SYS$RMS_GET
22:57:20.52 00018001 000604a0 SYS$RMS_UPDATE
22:57:20.52 -------- 00000080 SYS$SYNCH_INT
22:57:20.54 -------- 00000080 SYS$SYNCH_INT
22:57:20.54 000182aa 000604a0 SYS$RMS_GET
22:57:20.54 000182aa 000604a0 SYS$RMS_WAIT
22:57:20.54 00018001 00061a60 SYS$RMS_CLOSE
22:57:20.54 -------- 00000080 SYS$SYNCH_INT
22:57:20.56 00010001 00057680 SYS$RMS_GET
22:57:20.56 00018001 00065640 SYS$RMS_PUT
22:57:20.56 00010001 00057680 SYS$RMS_GET
22:57:20.56 00000001 SYS$EXIT_INT SYS$EXIT_INT
22:57:20.56 00010001 00065640 SYS$RMS_WAIT
22:57:20.56 00010001 00064080 SYS$RMS_CLOSE
22:57:20.56 00010001 00057680 SYS$RMS_WAIT
22:57:20.56 00018001 00054da0 SYS$RMS_CLOSE
22:57:20.56 -------- 00000080 SYS$SYNCH_INT
22:57:20.57 -------- SYS$EXIT_INT SYS$EXIT_INT
22:57:31.28 00000001 00000002 SYS$SETEXV
22:57:31.28 00000001 00000000 SYS$IMGACT
22:57:31.28 00000001 00000080 SYS$GETSYI
22:57:31.28 00000001 00000008 SYS$EXPREG
22:57:31.28 00000001 00000080 SYS$EXPREG
22:57:31.28 00000001 00000080 SYS$EXPREG
22:57:31.28 00000001 00000000 SYS$GETJPI
22:57:31.28 00000001 00000000 SYS$GETJPI
22:57:31.28 -------- 00000000 SYS$SET_PROCESS_PROPERTIESW



Of course that one-line is silly. It shoudl look like:

$ perl ssl_arg.pl < n.x
$ type ssl_arg.pl

#
# Sample output from ANALYZE/SSL/NORMAL/ACCES=U
#
# SYS$RMS_CONNECT sts: 00010001 acmode: U !22:57:20.42
# image: RMS+000e44d0 argct: 03
# arg 1:0000000000057680 2:0000000000000000 3:0000000000000000
# entry number: 00000188 number at completion: 00000188
# cpu id: 001 kernel thread ID: 0000 Pthread ID: 1
#
use strict;
use warnings;
my ($name, $status, $time, $image);
$image = ""; # Current SSL mostly shows Callee, not Caller :-(
while (<>) {
if (/^(\S+)\s+sts: (\S+).*!(\S+)$/) {
$name = $1;
$status = $2;
$time = $3;
$_ = <>; # Next line
# m/\s(\S+)\+/; # Locate Image Name
# $image = $1;
$_ = <>; # Next line
m/(.{8}) 2:/; # Match first argument value
printf qq(%s %s %s %-20s %s\n), $time, $status, $1, $name, $image;
}
}

[Ben Armstrong]

David & Hein, thanks to both of you. Your comments and sample code were very helpful. While the current problem focuses on the VMS port of Ruby, we also have to solve this for C++ on Windows (and possibly Linux as well). So, given what we've learned so far, we'll be looking next at flock(). If we get stuck, I'll know who to come to. :)