Stream_LF file shared access using standard I/O functions

[Sergejus Zabinskis]

Hi,
I have following problem:
I have several processes writing to one file. Writes are guarded with exclusive mutex. The program looks like:
const int MAX_MRS = 10000;
FILE *fd = fopen(fname, "a+b", "ctx=rec", "shr=get,put");
...
for(;;)
{
....
mutex.Enter();
fseek(fa, 0, SEEK_END);
if(bufsize > MAX_MRS)
write by MAX_MRS blocks using fwrite;
else
fwrite(buffer, bufsize, 1, fd);
fsync(fileno(fd));
mutex.Leave();
...
}

Creating file in this way I get with dir/full:
...
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 15390, Extend: 0, Global buffer count: 0, No version limit
Record format: Stream_LF, maximum 0 bytes, longest 10000 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWE, Group:, World:
Access Cntrl List: (IDENTIFIER=JUP_TEST1,ACCESS=READ+WRITE+DELETE+CONTROL)
(IDENTIFIER=JUP_TRAINING,ACCESS=READ+WRITE+DELETE+CONTROL)
Client attributes: None
...

After that, when I use this file I expect that all records (saved with fwrite call) must be followed by LF, but it is not true. Some of them (very small amount) not!!

Can somebody explain me what I am doing wrong.
Thank you
Sergejus

[Antoniov.]

Hi Sergejous,
before of all, Welcome to VMS forum :-)

A little quote: fseek use fa pointer not fd; I think it's typo. Also fsync().

Now

I expect that all records (saved with fwrite call) must be followed by LF

Why?
You open file in append binary mode (a+b). Binary mode tells to runtime "write record as is". And fwrite writes # items of declared len onto output file without formatting. Is there, at end of buffer the LF?
I'm not sure I get your real intention. Usually LF means end of line. If you want add EOL you have to open file in append standard mode and use fprintf function. In this way runtime add CR+LF at end of line. If you want to add only LF, open file a+b, and use
fprintf (fd, "%s\x000A",buffer);
I hope understand what you want to do.

Antonio Vigliotti

[Sergejus Zabinskis]

Thanks for response, Antonio.
I agree with you that writing to file in binary mode doesn't append anything. This is binary mode "definition". But I say what I see.
I calculate file size before writing and after and I notice that file grows additionally with 1 byte for each fwrite call.
Then I open this file as:
FILE *fa = fopen(fname, "rb", "shr=get,put");
and see that this additional byte is LF = 10.
Sergejus

[Antoniov.]

Hi Sergejus,
I didn't read carefully your question, so I didn't see fopen() function.
You use DEC CC extention of fopen. However, readind cc help and HP documentation I understand what you want to do.

FILE *fd=fopen(fname, "a+b", "ctx=rec", "shr=get,put");
where
ctx=rec means you write per record, overriding binary declaration; see here for detailed information http://h71000.www7.hp.com/commercial/c/docs/5763pro_024.html#fab_rab_keywords_1
I don't know how these option work. I guess you can use ctx=stm instead or you can declare rfm=stm or rfm=stmlf, but I'm not sure about this.
Don't forget, run time add LF at end of record even when you write splitted record (bufsiz > MAX_RMS).

Antonio Vigliotti

[Wim Van den Wyngaert]

RMS want to see records. You gave binary data and at the end RMS want to see a record terminator (LF). You didn't specify it, so RMS added it. Follow the advice of Antonio.

Wim

[Sergejus Zabinskis]

Hi again Antonio,
I can't use "ctx=stm" instead of "ctx=rec", because I get wrong behaviour if several processes are writing to the same file.

At the bottom of the page you sent:
http://h71000.www7.hp.com/commercial/c/docs/5763pro_024.html#fab_rab_keywords_1

is said (exactly about my situation):

You cannot share the default HP C for OpenVMS stream file I/O. If you wish to share files, you must specify "ctx=rec" to force record access mode. You must also specify the appropriate "shr" options depending on the type of access you want.


Anyway, thank you for your attention.
Sergejus

[Wim Van den Wyngaert]

Isn't a record in your file the whole file if you don't have LF's ?

Wim

[Antoniov.]

Sergejus,
perhaps this code can work for you

FILE *fd = fopen(fname, "a+b", "ctx=rec", "rfm=stm", "shr=get,put");

ctx is rec ad HP c runtime require;
rfm declare stream I/O; you can also declare stmlf (stream with LF).
As I posted, I never used DECC extension, so I have not direct experience about that.

Antonio Vigliotti

[Wim Van den Wyngaert]

Antonio,

I guess the default is taken from the file and the file is already stream_lf. This would modify it to stream and add terminators cr and ff.

Wim

[Hein van den Heuvel · Accepted solution]

Is the data that you write binary, or just ascii. Specifically, can it have LF byte (value 10) perhaps as part of an integer or float? Such embedded terminator will create a fresh record boundary on future reads.

From the RMS documentation for $PUT to stream_lf: 'If the last byte in the buffer is not a terminator, RMS adds the appropriate terminator'.
So if the last byte happens to have value 10, then this will become a shorter record as no terminator is addedd.

If you really want binary data then you should consider UDF (User Defined) record format where you know how many bytes to read and write, but RMS sees no struture.
Or... if the objects all have the same size, then you could/should consider a 'Fixed Length Record file (RFM=FIX).

Finally, if this is not part of a portable solution then you have to ask what value the C RTL add and consider calling RMS directly.

hth,
Hein.

[Sergejus Zabinskis]

Hi,
Antonio in his earlier post and Hein are absolutely right. Appending LF to the end of each record solves all my problems - I made few experiments and it works.
Before that I had situation that is described in Hein response. RMS added terminating LF, but due to the large amount of data there was situations when chr(10) from data buffer (not terminator) occured at the end of partial record and during future read it disappeared from data - so I got corrupted data.
No I append LF to the end of each record - and everything seems OK.
Many thanks for your kind attention to my problem!!!
Sergejus

[Sergejus Zabinskis]

About "rfm=fix". Unfortunately, data buffer sizes to save are irregular, so it is not acceptable from disk space viewpoint.
Thanks once more and bye.
Sergejus

[Wim Van den Wyngaert]

Another warning :
One of the file types defined by RMS is an RMS-11 stream format file, corresponding to a value of FAB$C_STM for the record format. The definition of this format is such that the RMS record operation SYS$GET removes leading null bytes from each record. Because this file type is processed in record mode by the HP C RTL, it is unsuitable as a file format for binary data unless it is explicitly opened with "ctx=stm", in which case the raw bytes of data from the file are returned.

Wim

[Antoniov.]

Sergejus,
happy you solved!
I'm waiting here for your next question!

Antonio Vigliotti

[Ian Miller.]

Sergejus, can you assign points to the responses you have recieved. See
http://forums2.itrc.hp.com/service/forums/helptips.do?#33

for an explaination.