Record Terminator

[Henry_52]

Hello,

I converted OpenVMS flat file to HPUX flat file using Vbackup. But When I read converted files, I can't find record terminator.(It looks like one record.) How can I see record terminator after conversion.

Henry

Below is replied e-mail from BBC.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is not that Vbackup did not put the record terminator into the file but rather because on VMS there might not have been a record terminator that UNIX will recognize. On some files in OpenVMS, for example, the record begins with a 4 digit number that indicates how long the record is. Extracted to a UNIX system this would look like a single record.

Adding characters to a file to indicate the end of a record is not always what people need. Without the options set Vbackup does not modify the data being restored.

To make some files useable on UNIX, there has to be some additional conversion. Vbackup cannot know, for example, what the fields in a record should be. That would have to be done after the data is extracted.

[John Gillings]

Henry,

I don't know what Vbackup is. However, under OpenVMS "flat file" doesn't really mean much. What was the exact record format? Stream? Variable? VFC?

You may need to CONVERT the file to a more Unix friendly format before moving it to HP-UX. Exactly what format depends on what record terminator you want to see.

The e-mail you quote says much the same thing. One GUESS as to what you may want to do:

$ CONVERT/FDL=SYS$INPUT flatfile.dat unix.dat
RECORD
FORMAT STREAM_LF
^Z

This will remove any variable length record or VFC metadata and insert an LF character as the record terminator (or course, if your data also includes LF characters you might have a problem).

Now run your Vbackup on unix.dat

[David B Sneddon]

Henry,

Depending on the size of the files and the
connectivity between the systems you might
want to try C-Kermit, which tends to handle
"flat file" transfers between different
architectures.

Dave

[David B Sneddon]

Henry,

Just had another thought -- have you tried using
ZIP (without saving the VMS attributes) then
using "unzip -aa" to force the extraction to
treat all files as text?

Dave

[Dima Bessonov]

Henry,

John basically said it all, I just want to add a couple of comments. Unlike Windows and UNIX file systems which (afaik) always treat the contents of a file as a stream of records divided by record terminators (usually , , or , depending on the system), VMS's RMS (aka Record Management Services) can create files using different record types as well. You can use DIR/FULL to see what record type the file(s) of interest has. Normally, if an application reads file record by record (i.e. explicitly or implicitly uses RMS), it doesn't see any difference between record types because it's RMS that takes care of the file internal structure. However, if an app bypasses RMS and reads raw data from the file block by block, it should learn the record type from the file header. Otherwise, it'll end up mixing actual record data with the file system control data. That's probably what happens with your Vbackup. Probably you can't do anything about the Vbackup itself but, as John already said, you can convert the files to the record type that is assumed by Vbackup. Just know exactly, what particular type of record terminator it wants. John's example applies to the case where single is needed. If you need , type "STREAM_CR" instead of "STREAM_LF", if it's , type just "STREAM". Hope that helps.

[Hein van den Heuvel]

> I converted OpenVMS flat file to HPUX flat file using Vbackup

Sounds like Vbackup is a tool on HP-UX.
Most readers here will not be able to judge whether this is the right or wrong tool.
Please provide use with quick pointer to the tool spec IF you are convinced this is the rigth solution.

More likely you were already 'too late'.
The 'damage' / confusion was probably introduced during the transfer and no tool on hpux can fix it afer the fact (unless your data has recognizeable record markers or a predictable (fixed) length.

FTP -- TEXT (ASCI) mode is the easiest 'conversion' during the transfer.
CHeck out the earlier replies on how to prepare a file on VMS (stream_lf) before the transfer if needed.

If none of the above replies help, then you'll have to communicate better what problem your are trying to solve.
What does the input look like (DIR/FULL? ANAL/RMS? DUMP/BLOC=COUN=1 ?)
What should the output on jUX look like? (od -x ?)

hth,
Hein.

[Antoniov.]

Henry,
typing DIR/FULL you can see file type. The unique flat file is fixed that's is treated as binary.
All other file type are not supported by other OS.
Text file could be easly converted but VFC or variable len file need conversion in VMS environment than may be exported to other OS.
Because conversion analyze is too complex you could post original file type for furthermore help.

Antonio Vigliotti

[Henry_52]

Vbackup restores OpenVMS backup tape to HPUX system directly. I have so many OpenVMS backup tapes and I won't use OpenVMS System soon. I restored OpenVMS backup tape to HPUX system and when I read restored flat file, I couldn't find record terminator.(Using vi, I met "line too long" error message).
Is there any solution after restoring tape to HPUX ?

Thank you for your quick replies.

Henry

Below is Vbackup homepage.
~~~~~~~~~~~~~~~~~~~~~~~~~~

http://www.bosbc.com

[Antoniov.]

Hi Henry,
I've looked into bosbc web site and it seems vbackup keeps file attributes.
Because you post vi error line too long, I guess you will edit some text files.
I think you wil try edit some VFC file instead text file. VFC and/or var record files in vms have a byte counter at begin of line and may does't exits end of line.
I hope follows example can help you to understand:

TEXT FILE (MSDOS AND VMS)
|M|Y|L|I|N|E|\r|\n| Text followed by CR (13) and LF (10)

TEXT FILE (UNIX)
|M|Y|L|I|N|E|\n| Text followed by LF (10) called NL in unix

VAR RECORD FILE WITHOUT TERMINATOR (ONLY VMS)
|06|M|Y|L|I|N|E| ASCII 6 to set record len

When you use vi in unix, in 1.st case you read text with \r and end of line; in 2.nd case you read fine; in 3.th case you receive line too long error because vi doesn't find newline (LF).
In unix there is a hex dump utility: you can use this command to view inside file and you can understand what's happening.


Antonio Vigliotti

[Martin P.J. Zinser]

Hello Henry,

from all the information I've found on the internet there is no indication that Vbackup does handle VFC files well. Your best bet is to get access to a VMS system, read your files there and rely on ftp and/or zip to do the "right" thing as far as record terminators are concerned.

Greetings, Martin

[Ian Miller.]

I think convert from VFC to the streamLF format that unix wants. Do this either on VMS or if no VMS system then after restoring files on to unix do convert there. You will have to write a convert program. See RMS documentation for details of record format (there is a record length word and some other stuff for VFC files. Watch out for the extra padding bytes inserted so that the two byte record length starts on a even byte offset).

[Hein van den Heuvel]

Henry,

My explicit for others implied request stands: What does the file actually look like on Unix? Just give us an 'od' HEX and/or ASCII dump for the first 1000 bytes or so in a TEXT file attached to your next reply here.

If the file you are opening is a 'plain' VMS text file, then it is VARIABLE LENGTH. That means each record is preceeded by a 16 bit binary length (NOT an 4-digit number, those are only used on Magtape) and starts at an even byte offset in the file. You can read such a file on NT or Unix with a perl script like:


binmode STDIN;
while (read STDIN,$length_word,2) {
# avoid using "S" or "V". just do the math.
($length,$null) = unpack ("CC",$length_word);
$length += $null*256;
last if ($length > 32767);
$read = read STDIN,$line,$length;
print "$line\n";
read STDIN,$null,1 if ($length & 1);
}


Now if the file you are looking for is a cobol report or DCL log file it my be this VFC others alluded to. In that case it gets much more tricky to interpret it. But as a first cut you could modify my program above to skip the first 2 bytes in the record (the ones following the 2 length bytes). That way you will get the raw data, but you will loose the formatting (double-space, over-punch, new-page)

So try that perl, and output a dump if you have no success.

Good luck,
Hein.

[Henry_52]

Thank you for your reply.
I executed "od -x" and I attached output file. Actually, I don't know about perl.
Please give me more detail information about perl command.

Thank you.

Henry

[Hein van den Heuvel · Accepted solution]

I'd highly recommend learning Perl as a generic data processing languange.
Those shell/DCL scripts that spend 90% of their code shuffling bits of string around amidst a few serious command executions (not countingn search/grep/awk/cut as serious) are proabably better in perl (or even awk).
The converse is also true: those perl programs executing 'system' every other line should perhaps haver remained a shell/dcl script.

Anyway... judging by the data of your file you are looking at a 'fixed length record file' probably with record size 120 (dec) bytes. Indeed, on VMS, there are no terminators in such file as they serve no purpose. VMS applications, through RMS, just get the data delivered in 120 bytes chunks.

Now if you move to a lower level OS ( ;-), you'll have to do that yourself. Just like VMS comes with an indexed/keyed records access method and no Unix has that build in.

Ooops, I'm getting further of topic :-)

Back to the script. Create a file 'split.pl' with the contents:

$length = 120;
binmode STDIN;
while (read STDIN,$line,$length) {
print "$line\n";
}

Now:

perl split.pl < vms.data > unix.text

That should do it (but I did not test)

You can also do 'one-liners' like:

perl -e 'binmode STDIN; print "$_\n" while (read STDIN,$_,120)' < vms.data > unix.text


Finally... one has to wonder that the heck you are doing trying to 'vi' a file for which you are so clueless about its contents.
IMHO you should have known it was fixed length, and have a pretty good idea about the record size and the fields within the records when you start poking at this file, based on application knowledge. That stuff has nothing to do with this being Unix or VMS.

Hope my script works well enough to get you looking in the right direction!
Good luck,
Hein.

[Henry_52]

Hi Hein van den,

Thank you for your detail information.
I executed perl script with vms file but I met Error message.
Please check below error.

Thank you.

Henry
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
syntax error in file a.pl at line 4, next 2 tokens "read STDIN"
Execution of a.pl aborted due to compilation errors.

[Hein van den Heuvel]

Well, I just did a cut from the forum and pasted the script into a unix file and it works for me (afte adjusting 120 to 12 for a simple test). If you re-typed instead of cut&paste then I'm afraid you fat-fingered it. Try again, check every character notably (, ), and ;
Try the one-liner also.
You'll figure it out. I know you will...

Hein.

[Martin P.J. Zinser]

Hello Henry,

tested Hein's suggestion on Unix. The code works fine. One thing to check, your error is on line 4, while the stuff it talks about is in Hein's example on line 3. Have you changed the program maybe? How do you execute it?

Greetings, Martin

[Henry_52]

Hi Hein, Martin

I tried several times but It's not working.
So I changed one thing and I think It's working now !! Thank you very much Hein.:)
Please check below and let me know if this is not good.

Regards,

Henry

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# script
$length = 120;
binmode STDIN;
while ( read (STDIN,$line,$length) )
{ print "$line\n"; }

$perl split.pl < vms.data > unix.text

# command line
$perl -e 'binmode STDIN; print "$_\n" while read (STDIN,$_,120)' < vms.data > unix.text

[Peter Hofman]

I think FTP-ing the file as type ascii from VMS to Unix should also do the trick.

[Martin P.J. Zinser]

Hi Peter,

FTP is an option if you still have your VMS system around. But if that is the case there are plenty of ways to deal with this. My reading of Henry's question was that he does not have this luxury and has to find a way to get data out of VMS backup savesets on Unix only.

Greetings, Martin

[Peter Hofman]

Hi Martin,

I must have missed that. I also missed the remark of Hein in the 5th reply about FTP. I have to start reading more careful I think :-)

regards, Peter