lseek() causing delays to shared file access

[John Thompson_10]

I have an application that has one or more processes reading from a common input file, each of these processes is running the same .EXE. Each file record contains the identity of the process that should handle that record: thus each file record is examined by all the processes but is written back to the file by no more than one of the processes (with a flag character reset to indicate that the record has been handled.

The problem is that file access conflict between the reading processes causes a delay that is longer than I would expect. The delay can be seen with 2 processes but can become greater than 10seconds with 4 processes. What is the cause of the delay and why should it be anymore than 1second ?

I have put together a program that shows the problem, see the attachment.

I usually compile /standard=vaxc, OpenVMS V7.3 on Alpha but can replicate the problem without the compiler switch and also on OpenVMS V8.2 also on Alpha.

To run the .EXE setup a foreign command and pass in the â input fileâ â delayâ and â identityâ e.g.
$ EDAG :== "$ww:edaget.exe;"
$ edag test1.log 1 s1

For the test program records in the input file are a single line of the form:

7#1#DEVA2#19-SEP#13:05#U#PROCESS##2#0#EDAPUT#DEVA2|FRED#18-SEP#10:41:54###47379##TEXT#FULL_TEXT|^^^^^^|Count 1,FORWARD,s1##

where the third character â 1â , separator â |^^^^^^|â and ID â s1â are essential.


The 'real' application also has a process that is creating the records in the file. I have a test program that replicates this function if any body requires it.

John

[Heinz W Genhart]

Hi John

I think you have a problem with a IAM .EVT File. The EDA Sender or dispatcher reads this files and forwards the events to a central appliance (for example)

What is the reason for reading this file with a own program?

One posibility would be to setup the dispatcher and let it execute some DCL actions (run your Program). If I could better understand your requirements, I could give better advise for it.

Regards

Heinz

[John Thompson_10]

Hi Hienz,

I am attempting make an improvement to the performance of the EDA sender.

I hope this answers your question.

John

[Craig A Berry]

Using lseek() on a record-oriented file is probably triggering a rewind followed by a reread of the entire file. I don't know that for sure, but unless the CRTL assembles and stores an internal map of RFAs to byte positions, takes the byte offset you pass and looks up what record you're on in a hash, then does a get by RFA, that's probably all it could do.

You'd probably be better off storing RFAs yourself and getting to where you want to be based on your own get by RFA, which of course involves delving a bit deeper into RMS calls than you are currently doing.

[Steven Schweda]

I'm with Mr. Berry. Using lseek() on a
non-stream file would seem to be asking for
trouble.

Knowing nothing about how any of this stuff
is implemented, I might guess (hope?) that
the suggested alternatives to lseek() ("help
crtl lseek"), namely fgetpos() and fsetpos(),
might do better, but I wouldn't bet on it.

[Hein van den Heuvel]

I don't know what the problem is, but I suspect the solution is to set RAB$V_WAT.
That is wait for a record lock to become available instead of polling.
Also set RAB$V_TMO + RAB$B_TMO as desired.

Looking at that code I would strongly encourage you to forgo the C IO if all you are doing is trying to co-erce it to have RMS do the right thing.

RMS knows very well how to append records to a shared sequential file. Just $PUT after a connect with RAB$V_EOF set. RMS will merrily return the RFA of the record just put if you care to know about that? Do you need to deal with adding multiple records as a block? There are ways to deal with that also (embedded terminators for stream type files, the undocumented SYS$MODIFY for other files)

The code as it stands seems so ugly. Notably the open code. The file is opened twice in a (feeble?) attempt to create two lock flavors and with with a (humongo?) risk of deadlock, not even counting the overhead. If you must, then CONNECT two RABs to one FAB. At least the buffers will be shared.
The second file is opened irrespective of the first file open status, hich then has to be unravelled, sort of, afterwards. Only try the second open if the first one succeeds?
The second open is allowed to create the file (O_CREAT), but if it needs to, then the first open would have failed no?

If you want to play with this then i would recommend SET FILE/STAT to see what RMS actions are done by the C RTL. Also, use ANAL/SYS... set proc xxx...SHOW PROC/RMS=(RAB) for the finer details on the RFA and such. Other insteresting RMS blocks: BDBSUM (buffer overview), FSB (those stats, if enabled), RLB (record lock blocks). Combine as SHOW PROC/RMS=(RAB,BDBSUM,...)

Also, do a simple SHOW PROC/LOCK.
Look for 'APPENDER'
The 8 byte locks are RFAs based locks under program control. The 4 byte locks are transient VBN locks managed transparently by RMS.

Finally, if you really want someone try this then please post the reproducer instructions as an attachment. It's rahter hard to decode in the original post.

Hope this helps,
Hein van den Heuvel
HvdH Performance Consulting

[John Thompson_10]

Thankyou to Craig, Steven and Hein for their comments.

I guess that I was hoping that there was some way to 'co-erce the C IO to make RMS do the right thing' aas Hein puts it. This would have been a quick fix with minimum impact. I think I already knew that the best way to get control of the file access would be to use the RMS function calls.

The code never writes new records, it only updates existing records, so I suspect that sys$update may be the one to use especially if sys$get will return an RFA. I'll have to read the documnetation.

Special thanks to Hein for his extra comments and tips.

John

[John Thompson_10]

I have rewritten the test program using sys$open, sys$connect, sys$close, sys$get and sys$update. Under my test conditions, where I saw the delay with the old .EXE the new one does not have the problem.

I'll now close this thread.

John

[John Thompson_10]

no further comment.