topic Re: Converting a fixed format file to stream LF in Operating System - OpenVMS

Converting a fixed format file to stream LF

Caroline Page — Tue, 20 Apr 2004 12:33:27 GMT

On OPENVMS Alpha I have a stream format file containing records length 277 which has been sent in physical records of 512 bytes ie FIXED 512 instead of fixed 277 or stream lf. I don't want to truncate the 512 byte records as the 277 byte records are wrapping round. I tried convert using streamlf and carriage return, but in addition to the correct carriage returns at the end of each 277 byte record I got an carriage return every 512 bytes.
How can I convert correctly to 277 byte records in stream LF ? - Many thanks - Caroline (more detail attached)

Re: Converting a fixed format file to stream LF

Hein van den Heuvel — Tue, 20 Apr 2004 13:46:37 GMT

I suspect the bytes in the file are correct on the disk. Only the file header describing hte file might be wrong. Try fixing with:

SET FILE/ATTR=RFM=STMLF

If that does not work then be sure include a text file with DUMP/BLOC=COUNT=2 for the transferred file and someone here will be able to deduct what to do next.
Basically... we'll look for a the "0A" hex record terminator. See whether it is there, where it is.

Note: Fixed length records of odd byte size (like 277) are stored with a filler byte in RMS fixed-length sequential file records, thus taking and even (278) number of bytes each.

Hein.

Re: Converting a fixed format file to stream LF

John Gillings — Tue, 20 Apr 2004 19:21:43 GMT

Caroline,

CONVERT can only process entire records. One record in corresponds to one record out. The most you can do is truncate or pad. As soon as you need anything that cuts and/or merges records, you need to write a program.

As Hein suggests, SET FILE/ATTR may fix the file (perhaps it was FTPed as BINARY instead of ASCII?).

If not, here's some DCL that might help. It treats the input file as a stream, cuts it into fixed length records and writes them to a stream_lf output file.

$ IF p1.EQS."" THEN INQUIRE p1 "Input file"
$ IF p2.EQS."" THEN INQUIRE p2 "Output file"
$ IF p3.EQS."" THEN INQUIRE p3 "Output record length"
$
$ ON WARNING THEN GOTO Cleanup
$ ON CONTROL_Y THEN GOTO Cleanup
$ OPEN/READ in 'p1'
$ CREATE/FDL=SYS$INPUT 'p2'
RECORD
FORMAT STREAM_LF
$ OPEN/APPEND out 'p2'
$ EOF="false"
$ READ/END=Cleanup in buffer
$ loop:
$ IF F$LENGTH(buffer).LT.p3
$ THEN
$ GOSUB more
$ IF eof THEN WRITE out buffer
$ ELSE
$ line=F$EXTRACT(0,p3,buffer)
$ buffer=buffer-line
$ WRITE out line
$ ENDIF
$ IF .NOT.eof THEN GOTO loop
$
$
$ More:
$ READ/END=eoin in line
$ buffer=buffer+line
$ RETURN
$ eoin:
$ eof="TRUE"
$ RETURN
$
$ Cleanup: SET NOON
$ CLOSE in
$ CLOSE out
$ EXIT

Re: Converting a fixed format file to stream LF

Willem Grooters — Wed, 21 Apr 2004 06:41:11 GMT

Caroline,

HOW did the file get formatted into fixed-size, 512-byte records? If you send a file bt PUTting it binary by FTP, this may have caused the problem.
I have had quite some problems with STREAM_LF before - being a non-native format it is quite obvoius that a lot of VMS programs and facilities cannot handle it properly without intervention, like CONVERT or the script John supplied. Because of that I would advice: if you can avoid using STREAM_LF to be processed by VMS-native programs, do so.

If you need the file to be copied from one system to another, try to wrap it up in some format, copy the wrapped file and unpack on the other side. You could use ZIP and UNZIP utilities (both are on the Freeware CD) but take care you may ned to specify certain options.

Re: Converting a fixed format file to stream LF

Martin P.J. Zinser — Wed, 21 Apr 2004 09:54:34 GMT

To followup on the Zip suggestion:

You do need zip "-V" (note the quotes and the case!) when creating the Zip archive.

Transfer binary, then do a plain unzip on the target system.

To get a full list of Zip Options do a zip -h.

Greetings, Martin

Re: Converting a fixed format file to stream LF

Antoniov. — Wed, 21 Apr 2004 10:56:42 GMT

Caroline,
may be a stupid idea but you can try.
Source file has not record attribute (carriage control = none) so source file is undelimited; if you set prn you could delimite end of record.
Syntax is:
$ SET FILE LRECL277.DAT /ATTR=(RAT:PRN)
then try to convert.

@Antoniov

Re: Converting a fixed format file to stream LF

Jan van den Ende — Thu, 22 Apr 2004 02:07:08 GMT

Caroline,

maybe it helpsto move one step further back.
Where did this file originally com from, and how did it get where you are having problems?
If my next guesses are incorrected, then the rest does not apply, but I suspect:
a. The original file was a VMS file
b. it was tranferred using FTP or RCP.

and probably c. via an intermediate non-VMS system.

If this is true, you should do a DIR/FULL on the original and resulting files, and every attribute that differs reset by
$ SET FILE/ATTRIB=...
for exact syntax $ HELP SET FILE/ATTR

hth

Jan

Re: Converting a fixed format file to stream LF

Caroline Page — Thu, 22 Apr 2004 08:23:44 GMT

Dear all - thanks for your helpful responses.

The file originally came from an AS400 - the original file has been deleted from that system and it is not possible to retireve it from archive or recreate it as the source data has changed.

Hein - thanks for your suggestion - setting the file attr did not solve the problem - but your point about the 278 storage was very helpful.

John - I tried the script - but got caught out by the max length of a dcl symbol which is 256 bytes - am currently working on a variation of your script - many thanks

Willem, Martin & Jan - you're all right about the problem being how the file was sent - but unfortunately as described above I can't get the source to resend .

Antoniov - thanks for the idea - will give it a try.

In the meantime I enclose a dump file of the first 4 blocks of the file (512 fixed version)
Thanks again - Caroline

Re: Converting a fixed format file to stream LF

Jan van den Ende — Thu, 22 Apr 2004 08:53:52 GMT

Caroline,

there mat be a way to get along with longer DCL symbols.
The current maximum length is about 1000 (1K - some overhead). btw, in VMS 7.3-2 it increases to 8K.

The functions available on symbols > 255 chars are limited.
Things like f$LOCATE and F$ELEMENT don't work.
However, you CAN do an f$extract.
This way you split it up in 100 or 200 byte chunks, process any logical parts off of then beginning, concat the next etc.
You cannot SHOW the long symbols, and you cannot simply write 'm.
However,
$ WRITE/SYMBOL file symb
WILL work.
And file may well be SYS$OUTPUT.

Yes, it is a little cumbersome, and it looks clumsy, but if you NEED it, you are glad there is at least a way. (I definitely have been, on various occasions).

hth

Jan

Re: Converting a fixed format file to stream LF

Hein van den Heuvel — Thu, 22 Apr 2004 11:21:01 GMT

> Hein - thanks for your suggestion - setting the file attr did not solve the problem - but your point about the 278 storage was very helpful.

This may sound obnoxious, but I think it did work. The dump clearly shows hex '0A' at the right intervals. Changing the attirbute to STREAM_LF will turn those chunks into records.

So please explain why you concluded it did not work.

Re-verify the SET FILE/ATTR with DUMP/RECOR=COUNT=3 (or however many you can stand).

Does you application handle STREAM_LF files?
IF not, after making the file useabel with SET FILE convert it. For example simply using CONVERT/STAT streamfile test/FDL=NL:

Good luck,
Hein.

Re: Converting a fixed format file to stream LF

Hein van den Heuvel — Thu, 22 Apr 2004 11:41:29 GMT

ooops, I send that too quickly.
I meant to inlcude I why I think the file looks good. I made a quick check list printing the hex value, and hex offset in the block for every 278 bytes (277 for the record, 1 for the terminator). It lines up great with the dump.

Hein.

> perl -e 'while ($i<10) {$x=278*$i++; printf ("%04x %04x\n",$x,$x%512)}'
0000 0000
0116 0116
022c 002c
0342 0142
0458 0058
056e 016e
0684 0084
079a 019a
08b0 00b0
09c6 01c6

Re: Converting a fixed format file to stream LF

Caroline Page — Thu, 22 Apr 2004 13:10:03 GMT

Hein - Not at all objectionable... it does line up on 278 - the dump I sent was from the original file - but I also get extra breaks ... I need to get rid of these without removing the 278 breaks

- as it stands when I attempt to load the file into Oracle , oracle objects very strenuously

I do appreciate your help very much.
Kind regards - Caroline

To aid comparison will attach the output from a type /pa from a correctly formatted file and from the original file as well as a dump of the file before and after the stmlf command and a dump of a correct file.

Thanks again - Caroline
...sorry if I've done something daft along the way....

Re: Converting a fixed format file to stream LF

Caroline Page — Thu, 22 Apr 2004 13:13:43 GMT

Jan - thanks for your very helpful hints - I was unaware of the write /symbol ... I shall amend my script and try again.

Kind regards -Caroline

Re: Converting a fixed format file to stream LF

Hein van den Heuvel — Fri, 23 Apr 2004 00:56:59 GMT

Well, I am still confused why the straigth SET FILE did not do the job. The dump seems to show that the fullsized records end in x0A as expected, the short ones without. That means (in my book) that the short ones should not show up as record, as there is no terminator.
Possible explanations:
- You inherited a 'no-span' attribute
- You used an extra, un-needed convert.

Note... because you happen to want an odd fixed length size you can actually also try SET FILE/ATR=(RFM=FIX,LRL=277,MRS=277)
As I wrote, RMS will round up, which happens to nicely 'eat up' the x0A line-feeds.

Anyway, it seems you are stuck with this full records + partial. Should be easy enough to glue them back together. Using AWK:

$ gawk /out=fixed /com="{l=length(); if (l<277){print}else{x=$0;getline; print x $0}}" broken

Using PERL:
$ perl -pe "if (length()<278){chop;$_.=<>}" < broken > fixed

(The 278 and chop are needed because perl instantiates a record terminator!)

Using DCL:

$if p2.eqs.""
$ then
$ write sys$output "Please provide input and outfile file names"
$ exit
$endif
$open/read p1 'p1
$open/writ p2 'p2
$loop:
$read/end=done p1 rec
$if f$len(rec).lt.277
$ then
$ read/end=done p1 some_more
$ rec = rec + some_more
$endif
$write/symb p2 rec
$goto loop
$
$done:
$close p1
$close p2

Oh, btw... there is one more, little known, fun tool for file conversions that comes with VMS. It has an odd name: EXCHANGE/NET
It has record blocking/deblocking. Check it out! (HELP)

Hein.

Re: Converting a fixed format file to stream LF

Antoniov. — Fri, 23 Apr 2004 01:59:37 GMT

Hello Caroline,
looking into your dump, I've seen after LF (0x0A) there are furthermore information; if I understand this is a valid data because it appear as same format of 1.st valid record; same for next logical record.
If is it, you byte are all sequentially stored in your file and any record end with new line or LF (0x0A); there is no extra bytes.
Your trouble, I think, is exchange file to other application; Hein's command would work fine SET FILE/ATTR=(RFM=FIX,LRL=277,MRS=277) then you can view every record with DUMP/REC.
Also you could try
$ SET FILE /ATTR=(FRM=STMLF,MRS=277,RAT:NONE)
so you file is managed as unix text and you could view file using TYPE.
I think you have no convert file.

@Antoniov

Re: Converting a fixed format file to stream LF

Caroline Page — Fri, 23 Apr 2004 11:01:57 GMT

Antoniov
thanks very much
- I did as you suggested and tried SET FILE/ATTR=(RFM=FIX,LRL=277,MRS=277) - this looked brilliant on the dump - but mysteriously only fixed the first 15844 records - as I discovered when I tried to load the file into oracle - HOWEVER - I think we hit the jackpot with your second suggestion SET FILE /ATTR=(FRM=STMLF,MRS=277,RAT:NONE) ! - I can't express how delighted I am !
Many many thanks to you and Hein and everyone who has contributed

Re: Converting a fixed format file to stream LF

Antoniov. — Fri, 23 Apr 2004 11:06:54 GMT

Caroline,
happy you solved.
If you'll need again post another thread :-)

@Antoniov