topic Re: DEC C RTL skips some bytes when reading binary files in Operating System - OpenVMS

DEC C RTL skips some bytes when reading binary files

Andreas Freiherr — Tue, 07 Feb 2006 14:16:57 GMT

Hello specialists,

I am trying to read binary files, e.g., a .BMP file saved to VMS from a PC using PATHWORKS. I am doing fine (i.e., can successfully process a whole file) unless I hit a byte with a value of 0x0B: after this byte, I loose synchronization with the data stream by missing zero bytes. This happens with fgetc as well as with fread, so I am afraid the problem may be somewhere deep inside the RTL - or, am I doing something wrong?

I am using...

HN01$ cc/ver
DEC C V6.0-001 on OpenVMS Alpha V7.3-2
HN01$

(Too old? - If you can point me to a more recent version that fixes my problem, I'll happily consider this a valid and good answer.)

Please compare the output of my attached C program that dumps the first 64 bytes of a file once using fgetc and once using fread...

HN01$ r readbin
42 4d 3e 1a 00 00 00 00 00 00 3e 00 00 00 28 00
00 00 fb 00 00 00 d0 00 00 00 01 00 01 00 00 00
00 00 00 1a 00 00 13 0b 13 0b 02 00 00 00 00 00
00 00 00 00 00 00 ff ff ff 00 ff ff ff ff ff ff

42 4d 3e 1a 00 00 00 00 00 00 3e 00 00 00 28 00
00 00 fb 00 00 00 d0 00 00 00 01 00 01 00 00 00
00 00 00 1a 00 00 13 0b 13 0b 02 00 00 00 00 00
00 00 00 00 00 00 ff ff ff 00 ff ff ff ff ff ff
HN01$

... to the (partial, but corresponding) output of a DUMP/BYTE command:

00 28 00 00 00 3E 00 00 00 00 00 00 1A 3E 4D 42 BM>.......>...(. 000000
00 00 00 01 00 01 00 00 00 D0 00 00 00 FB 00 00 ..Ã»... ......... 000010
00 02 00 00 0B 13 00 00 0B 13 00 00 1A 00 00 00 ................ 000020
FF FF 00 FF FF FF 00 00 00 00 00 00 00 00 00 00 ................ 000030

(To create a similar file for your own test, save a .BMP with a resolution of 72 dots per inch. This will be stored as 2,835 dots per meter, which is 0B13 in hex.)

Any ideas? - Thanks in advance!

Andreas

Re: DEC C RTL skips some bytes when reading binary files

Hein van den Heuvel — Tue, 07 Feb 2006 14:30:48 GMT

What is the file record format? DIR/FUL
or WRITE SYS$OUTPUT F$FILE(file,"RFM")

I'm guessing is is STM instead of STMLF.
The 0B = VT would then be a record terminator.
Try SET FILE/ATTR=RFM=UDF

hth,
Hein.

http://h71000.www7.hp.com/doc/731FINAL/4523/4523pro_007.html#rms_record_format_field

FAB$C_STM
Indicates stream record format. Records are delimited by FF, VT, LF, or CR LF, and all leading zeros are ignored. This format applies to sequential files only and cannot be used with the block spanning option.

Re: DEC C RTL skips some bytes when reading binary files

Hein van den Heuvel — Wed, 08 Feb 2006 08:38:32 GMT

Wel..... Inquiring minds want to know...
Was the file in question marked a 'STREAM' or 'STM'?

That is a nice name for a nasty file format.
It was really is only there for a compatibilty with RSTS for which the need expired more than 2 decades ago (imho).

The only RMS record format which comes closed to stream-of-bytes is the 'UNDEFINED' or UDF format. Unfortunately some generic tools barf on this files, instead of just taking a buffer full at a time and deal with it.

Therefor one is often forced to claim 'STREAM_LF' and the next best, but not entirely correct description of the record format.

hth,
Hein.

Re: DEC C RTL skips some bytes when reading binary files

Andreas Freiherr — Wed, 08 Feb 2006 09:31:03 GMT

Hein,

You're right, the record format is STM. However, does this matter when the file is opened for BINARY access (my demo program says fopen ("test", -->"rb"<--, "dna=.bmp"))?

You're of course right, 0B is the vertical tab control character, but I know I am able to read other terminators (CR, LF, etc.) without any problem.

Also, remember that the VT gets through, but I am loosing the following NULs, even though I successfully read several NULs before from the same file!

I tried fread with the complete header declared as a struct, and lost the same bytes from the file. In this case, no *character* processing should be involved, right?

If terminators like these cannot be read this way, then what do I need to do instead? - It might be interesting to see how PATHWORKS reads this .BMP file back when I open it from the PC, because this *does* work.

Andreas

Re: DEC C RTL skips some bytes when reading binary files

Andreas Freiherr — Wed, 08 Feb 2006 12:42:31 GMT

Hein, Everybody,

after some off-forum communication, I conducted some additional tests with different record format settings on the same file, in which I had stored different sequences that might cause effects, involving NULs (ASCII 00), LF (ASCII 10 = 0A hex), VT (11 = 0B), FF (12 = 0C), and CR (13 = 0D):

0A 00: if stored in file, is read as...
... 0A only with STM,
... 0A 00 with STMCR,
... 0A 00 with STMLF, UDF, and VAR records

0B 00: similar to 0A 00

0C 00: similar to 0A 00

0D 00: read as...
... 0D 00 with STM
... 00 only (CR dropped) with STMCR
... 0D 00 with STMLF, UDF, and VAR records

0D 0A 00: read as...
... (nothing, all 3 bytes dropped) with STM
... 0A 00 (CR dropped) with STMCR
... 0D 0A 00 with STMLF, UDF, and VAR.

All tests were run with "rb" as the mode in fopen.

This tells me that line terminators are handled even if binary access is requested. I am tempted to consider the "b" in the mode string a no-op.

The STMLF, UDF, and VAR formats behave identically (surprised me), except that the record length field is hidden when VAR is used (no surprise).

Another observation is that suppression of leading NULs occurs only with certain record formats and is not directly related to recognition of a line terminator.

There was no visible difference in behaviour between fgetc and fread (so what good is fread, other than saving me coding a loop?)

To me, these results don't make sense, so it's probably best to rewrite part of my program, following Hein's suggestion and using native VMS I/O routines. Bye, bye, portability!

Re: DEC C RTL skips some bytes when reading binary files

David Jones_21 — Wed, 08 Feb 2006 21:44:27 GMT

The problem is that the C RTL opens any non-STMLF file by default in record mode, so RMS is gobbling the data before the fopen mode ("rb") applies. Try adding a "ctx=stm" as an additional, argument to the fopen() call.
It's still non-portable, but it is less involved that writing separate routines.

Re: DEC C RTL skips some bytes when reading binary files

Andreas Freiherr — Thu, 09 Feb 2006 14:05:11 GMT

David,

You're absolutely right: "ctx=stm" does the trick.

And, for portability, it's far easier to toggle the presence of this argument in fopen than to use a layer of I/O routines on top of the RTL or RMS.

Case closed, thanks!

topic Re: DEC C RTL skips *some* bytes when reading binary files in Operating System - OpenVMS

DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

Re: DEC C RTL skips *some* bytes when reading binary files

topic Re: DEC C RTL skips some bytes when reading binary files in Operating System - OpenVMS

DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files

Re: DEC C RTL skips some bytes when reading binary files