VMS adding bytes

[owilliams]

A user is copying VMS files sequentially to another system and he says that VMS is adding bytes to the files. I am not yet sure of the application he is using yet, but any ideas you guys have would help me tremendously. Thanks in advance.

[Heinz W Genhart]

Hi owilliams

of what type is the "another system" (VMS, UNIX, ???)

How do they copy the file?

Regards

Heinz

[owilliams]

I am waiting for the user to respond. And I will post the answer. Thanks.

[Hein van den Heuvel]

VMS is not adding bytes, they are incorrectly transferring the file.

'normal' VMS files are veriable length with a 2 byte record length pre-fix and optional alignment byte for odd sized records.
If one errorneously transfers such file in BLOCK = BINARY = IMAGE mode, then those bytes may appaer to be 'added bytes'. I'd like to argue they are not added, just misunderstood.

Solution?
- transfer in RECORD = ASCII mode.
or
- pre-convert to a target compatible mode.
For Unix that would typically be STREAM_LF, for Windows STREAM.

Example for OpenVMS 8.3:

$CONVERT/FDL="RECORD; FORMAT STREAM" old new

Good luck!
Hein.

[owilliams]

I just talked to the user. What it comes down to is he is attempting to create a totally flat file in VMS. How do you do that?

[Hein van den Heuvel]

For 'flat files' which are just a stream of bytes most applications typically use a 'fixed lenght record 512 byte' file.

The 'right' solution is to use an file with record format = undefined
Check the RMS manuals and HELP SET FILE/ATTR

This might NOT be easy, and will notably depend on the language used. Some languages try to 'help' too much.

If the file exists today already, but as variable length, then convert can help as indicated before.

It's about time you shared a
- $DIR/FULL for the file,
- The OpenVMS version
- The desired language
- The method used to write the file (open statement, write statemnt)
- The desired method to transfer the file. (FTP? NFS? Samba?)

Hein.

[owilliams]

I'll get that info.

[owilliams]

I am working on the file info and language.
OpenVMS version - V7.2-2 AlphaServer GS80 6/1001
Using a write statement.
FTP to transfer the file.

[owilliams]

I am working on the file info OpenVMS version - V7.2-2 AlphaServer GS80 6/1001
Using a write statement.
FTP to transfer the file.
The language is Basic.

[Hoff]

Sequential files are not necessarily compatible across platforms.

You're very likely getting nailed by the record organization inside the file, or by the lack of implicit conversion during the transfer.

I generally use stream LF files for transfers, as these are compatible with Unix systems.

With Windows, Stream tends to be the best choice, but there are a number of Windows tools that can and do deal with Stream LF. (Notepad does not. Wordpad does.)

VFC files will get you extra bytes embedded in the file.

The easiest way is to either use Stream LF files, or to convert the VFC file into stream LF as part of the the transfer via the COPY/FTP command. (COPY/FTP is easier to use and easier to embed in a DCL procedure than is the FTP utility.) To convert, the usual approach is COPY/FDL.

Some tools will implicitly perform the conversion. See the ADF file stuff within NFS, for instance.

I've previously posted up details of the benefits of COPY /FTP at the new web site, I'll post up file conversion and file transfer details for the site and for the FAQ. For whatever reason, this question is arising with some regularity of late.

Stephen Hoffman
HoffmanLabs LLC

[Hein van den Heuvel]

>> I'll get that info.

Why is the person itself not communicating here? No disrespect intended but what value are you adding to this process?


>> Using a write statement.

Basic does not have a write statement.

>> FTP to transfer the file.

I would highly recommend just creating a bunch of file of various flavors using CONVERT and an FDL file (with 7.2 you can not use the FDL string), transfer them and see what works. Then create the program to write what worked.

>> The language is Basic.

Ah, that's actually tricky, because basic is picky (with respect to file types). It has an 'organization undefined' clause but somewhat confusingly that does NOT map to Record Format = Undefined. One will need a USEROPEN to force that issue if that path to a solution is chosen. Example below.

I would recommend to use the fixed length 512 byte record method....

or... use the convert.

Was the convert suggestion tested?

Why not?

$CREATE udf.FDL
RECORD; FORMAT UNDEFINED
$CONVERT/STAT/FDL=udf old.dat new.raw
$FTP...


Regards,
Hein van den Heuvel

[Hein van den Heuvel]

oops, forgot the undefined record example....


$ type test.bas

call lib$init_timer
map (x) string test = 1000
open "test.tmp" for input as file #1, useropen rfm_udf, &
organization undefined, recordtype any, access append, map x, extendsize 1000

test = "organization undefined, recordtype any, access append"
for i% = 1% to 10%
for j% = 1% to 1000%
put #1, count j%
next j%
next i%
call lib$show_timer
end

$ type useropen.bas
FUNCTION LONG rfm_udf (FABDEF FAB, RABDEF RAB, LONG CHANNEL)
OPTION TYPE = EXPLICIT

%INCLUDE "$FABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET.TLB"
%INCLUDE "$RABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET.TLB"
EXTERNAL LONG FUNCTION SYS$CREATE, SYS$CONNECT
DECLARE LONG stat
FAB::FAB$B_RFM = FAB$C_UDF
RAB::RAB$B_MBF = 4%
RAB::RAB$B_MBC = 120%
RAB::RAB$L_ROP = RAB$M_WBH
STAT = SYS$CREATE(FAB)
STAT = SYS$CONNECT (RAB) IF (STAT AND 1%) = 1%
rfm_udf = stat
END FUNCTION

[Jon Pinkley]

You stated that FTP was being used to transfer the files. What is on the other end of the transfer, and which end is initiating the transfer (the client)?

ASCII mode is the default, unless you are attempging to use something like Windows IE, which seems to use binary mode for everything.

If it is a windows PC that is getting the file from the VMS system, then try a different client program on the Windows PC. For example, the command line client that you run from a cmd prompt.

If this is a Win XP system, click start, click RUN, enter "CMD", ok, and you will get a command prompt window.

C:\blablabla> ftp vmshostip
username: whatever
password: whatever
ftp> ascii ! this is the default, not really needed
ftp> get VMSfile localfile
ftp> quit

ftp in ascii mode should handle the translation of records to records fine. As long as the record lengths do not exceed the limits of either end, files will be transferred a record at a time. You can even transfer an indexed file on VMS to a text file on the other end.

My point is, whether it is "stream_lf", var carr control, vfc, etc, makes NO DIFFERENCE if the ftp mode is ASCII.

That isn't true for scp, sftp, binary mode ftp, etc.

Attached is a demo going from vms to unix showing that the internal representation of "records" is automatically handled by ASCII mode ftp.

[Hein van den Heuvel]

Jon, I suspect that they specifically do NOT want to use ascii /record mode.

That is my interpretation of "he is attempting to create a totally flat file in VMS. "

Like a 'byte stream' file, possibly with binary characters and possibly without a single linefeed values byte which would make RMS unhappy if it was labelled stream_lf and larger than 32KB.

I suspect they will consider every linefeed (\n) to be an added byte.

Hein.

[Jon Pinkley]

Hein,

I agree, we need more info, and the term "flat file" means different things to different people.

Without better description of what the problem is all we can do is make SWAG responses.

One other possible scenario is the use of "fixed" length files with odd byte lengths. RMS imposes word (16 bit) alignment on the start of new record for fixed length files. In other words an RMS file with fixed length 1 byte records will consume the same amount of space as a file with the same number of 2 byte records. To create odd length fixed records, write pairs of odd length fixed records into a single RMS record that is twice as large. Then transfer the file in binary mode.

Example: two 19 byte "logical" records are put end to end into a single 38 by RMS record. Then the file header has the meta data describing the records, and the data blocks of the file have only data, no record delimiters. Similar to synchronous (no start/stop bits) vs. asynch data transmissions.

I've attached a demo showing that 256 records in a 1 byte fixed format file take the same space as 256 records in a 2 byte fixed format file.

The reason for my previous response was that
I've been seeing a lot of recommendations in this and other ITRC threads to "convert to stream_lf, etc." before transferring with ASCII mode ftp, and that just isn't a requirement. It won't hurt, and it can be helpful when not using ASCII mode transfers, and it may indeed help when using a lame ftp client (like windows IE), but it should not be necessary.

FTP has many warts, but is does have some features that newer transfer protocols lack. In that respect it is similar to Kermit, which could transfer files between more types of systems than for example x, y or zmodem.