Fixing a RMS Indexed file that has been FTP'ed

[DanMG]

I have an OpenVMS RMS indexed file that was FTP'ed to a Windows network then stored off on their archiving system. This is the only copy of this file that I have. Yes, I know this is a no no... But is there any way to recreate it into a VMS indexed file again? Where do I find any doc on RMS indexed file internals, if I wanted to do this? Or do I tell my auditors... "No it is impossible" as everything I have read so far.

Thanks for any input that anyone has on this! It is much appreciated.


Thanks
Dan

[DECxchange]

Did you do a binary transfer on it? Another idea would be to zip the file first on VMS, then binary transfer it to the PC.

You might want to check the VMS RMS Guide, the Convert Utility, and the FDL Language Editor manual for info on how to convert from indexed to other formats.

At the VMS prompt, you can type $ help convert file
then /fdl

Or
help analyze/fdl

Probably, I would to an analyze/fdl on the VMS end before sending it. That way you have an FDL record of the file structure.

But come to think of it, I don't know if PCs have anything like indexed files on VMS, so maybe it's not a good idea.

Maybe you should just do a VMS backup to tape or another disc and store it that way.

[Robert Gezelter]

Dan,

It depends upon how it got corrupted. If it was FTP'ed in binary mode, then it might just be missing attributes. If it was transferred in text mode, it would probably need more work.

The important question is how much are the contents worth, and how bad is the damage. Some of the RMS format information is in the RMS manuals, some of it is in various books, and some of it is in various presentations that have been made over the years.

The first step is to see just how bad the damage is. The second step is to determine if we have a known file with the same setup (keys, formats, etc.) so that it can be used as a model.

If you have never delved into this area before, you may want to retain someone who has experience in this area [disclosure: we provide services in this area, as does Hein and several other regular contributors to this forum).

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

DECxchange,

Actually, if the question were "How do I backup an indexed file to archive it from a Windows platform?", the answer would be to first use the OpenVMS version of ZIP with the "-V" option to preserve the OpenVMS file attributes.

This pre-supposes that the file will fit within the limits of ZIP, and that the resulting file is transferred correctly to the Windows platform. For safety, I always try to do a TEST operation on the ZIP archive on the PC side. I also frequently "roundtrip" the file back to the OpenVMS platform and do an UNZIP to confirm that the process is reversible. Paranoia on this type of issue can be a very safe thing.

However, the question was "How do I recover a file that was transferred to a Windows machine in the past".

- Bob Gezelter, http://www.rlgsc.com

[John Gillings · Accepted solution]

Dan,

This isn't quite converting "Hamburger into Cow" - you've got a reasonably good chance of recovering something. It would help A LOT if you know the attributes of the original file. Even just a DIRECTORY/FULL command of the original! In particular, I think you need to know the RFM and RAT (you already know the ORG).

In future, a good way of protecting a file from the ravages of other operating systems is to put it in a BACKUP saveset. The attributes of a saveset can always be reconstructed easily (as of V8.3 there's even a BACKUP/REPAIR).

Here's a simple example of recovery which might help you. I took a copy of this file and FTPed it to a PC:

INDEXED.DAT;1 File ID: (1132,100,0)
Size: 42/42 Owner: [JG]
Created: 15-FEB-2008 12:09:51.82
Revised: 15-FEB-2008 12:09:51.94 (1)
Expires: 16-FEB-2008 12:09:58.43
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Indexed, Prolog: 3, Using 3 keys
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 42, Extend: 0, Maximum bucket size: 2, Global buffer count: 0
Version limit: 20
Record format: Variable length, maximum 0 bytes, longest 0 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

When copied back it looked like this:

INDEXED.BACK;1 File ID: (1134,48,0)
Size: 42/42 Owner: [JG]
Created: 15-FEB-2008 12:10:13.18
Revised: 15-FEB-2008 12:10:13.18 (1)
Expires: 16-FEB-2008 12:10:13.18
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 42, Extend: 0, Global buffer count: 0
Version limit: 20
Record format: Fixed length 512 byte records
Record attributes: None
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

I then "fixed" the attributes thus:

$ set file/attr=(org:idx,rat:cr,rfm:var) indexed.back

resulting in:

INDEXED.BACK;1 File ID: (1134,48,0)
Size: 42/42 Owner: [JG]
Created: 15-FEB-2008 12:10:13.18
Revised: 15-FEB-2008 12:12:47.37 (2)
Expires: 16-FEB-2008 12:12:47.37
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Indexed, Prolog: 3, Using 3 keys
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 42, Extend: 0, Global buffer count: 0
Version limit: 20
Record format: Variable length, maximum 512 bytes, longest 512 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List: None
Client attributes: None

And ANALYZE/RMS is happy with it.

Check RMS File Integrity 15-FEB-2008 12:16:21.16 Page 3
DISK_USER0:[JG]INDEXED.BACK;1


Segment Positions: 16 0
Segment Sizes: 1 15
Data Type: string
Name: ""
First Data Bucket VBN: 16


The analysis uncovered NO errors.


ANAL/RMS INDEXED.BACK

and the data looks correct.

Worst case you could try different combinations of RAT and RFM - but if it's fixed length records, you'll have to figure out the length.

[John Gillings]

Dan,
Sorry, just noticed my output file might not be quite right:

>Record format: Variable length, maximum 512 bytes, longest 512 bytes

In my case all the records are smaller than 512 bytes, so MRS:512 doesn't matter. You may need to specify a higher MRS to fit in longer records. The LRL doesn't matter, as it can easily be fixed with a CONVERT of the final file.

To match the original file (including MRS and LRL), I should have used the command:

$ set file/attr=(org:idx,rat:cr,rfm:var,lrl:0,mrs:0) indexed.back

Finally, a CONVERT to make sure everything is clean and tidy:

$ CONVERT INDEXED.BACK INDEXED.RESTORED

[Hein van den Heuvel]

>> Fixing a RMS Indexed file that has been FTP'ed

Now what possed folks to do that?
Anyway... BINARY or ASCII?
RAW, binary data in the records, or just text?

Start with just FTPing is back in BINARY mode. Then $DUMP/BLOCK=COUNT=6 and attach that output as a .TXT file to a reply, if further help is neeed.

So how do you know it was indexed?
Do you still have an FDL?

But is there any way to recreate it into a VMS indexed file again?

Quite possibly.

John outlined on technique, flipping bits in the file header, but that tends not to get them all.
I would suggest to find an FDL (from a different box with the same application?), and $CREATE/FDL followed by $COPY/OVER ftp-file new-indexed file. Now try use the file.

>> Where do I find any doc on RMS indexed file internals

You probably will not need that. But it should not hurt to check out the manuals, and in particular:
" Guide to OpenVMS File Applications"
Specifically in indexed file chapter:
http://h71000.www7.hp.com/doc/731FINAL/4506/4506pro_005.html#021_indexedfileorganization

The best way to learn about indexed file internals is to use ANAL/RMS/INTERACTICE on a working file with understood data, like a copy of SYSUAF or sys$system:VMSMAIL_PROFILE.DATA and drill down, trying to take as little as possible for granted.

The on disk bits are defined in SYS$LIBRARY:LIB.MLB (BKTDEF, IRCDEF,...)

Yo may find that my old slideset help, notably the 'patching' section:
http://h71000.www7.hp.com/freeware/freeware60/rms_tools/rms_tuning.ppt

>> do I tell my auditors... "No it is impossible"

If that's what you gets you of the hook, then why not!?
The better answer is 'sure, but it is going to cost you'. Time to pay up for lack of education and training earlier. What is it worth to them / the company?
For a rubber stamp? Just say no.
For a challenge! Sure.

Anyway, if the file was FTP'ed BINARY, then the ods are pretty good. And if it was transferred ASCII,a dn the contents was text, then the odds are good, as you will just have a sequential file, which is possibly good enough already.
Your ods will increase with the availability of a similar file and/or a matching FDL file.

For mere money I'll be help you assess whether it can be done, and possibly do it!

Good luck!


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[DanMG]

You all are awesome!

It was transfered in BINARY and I have an FLD on my file as it is a reoccuring monlthy file created by my system.

The commands I ended up doing were:
$ SET FILE/ATT=(RAT:CR,ORG:IDX,rfm:fix,lrl:140) dan.work
$ convert /fdl=knowen.fdl dan.work good.file

Thanks much to each & every one of you...

[Jon Pinkley]

Dan,

I see you have solved your problem, you probably could have used dan.work directly.

At any rate, here's what I had already written, and someone might find the info useful for the transferred in ascii mode section.

Since you are doing this on an ongoing basis, you should change the procedure so you don't have to do this in the future.

Backup of the file to a saveset followed by zip "-V" will give you the "supported" way to archive, or you can just use zip "-V" directly. It depends a bit on what you are most familiar with, whether you want to save multiple versions, etc. Zip 2.32 works for every indexed file I have tossed at it, so I would be inclined to use that instead of the interermediate backup saveset.

Jon
------------------------------
When the file was transferred to the PC using FTP, information was lost. Specifically the metadata about the file.

Since the file has been transferred twice once to the PC, once to the archival system, it is possible that the second transfer did additional modification.

There are two basic ftp transfer modes, image (binary) and ascii (record oriented).

Was the file ever used on the PC, or was it just put there for archival purposes? If it was actually used for something, then my guess is that the file was transferred in ascii mode, and you have a file with no index structure left, but the data is probably all there, or at least it was immediately after the file was transferred to the PC.

If it was transferred in binaray mode, then the file on the PC wouldn't be usable (at least not by standard windows utilities), but the index structure would still be there, but the metadata from the file header would be gone. However, the internal index structure of the file should still be intact.

If you know what created the file on VMS, and you can create an "empty" file with all the index key structure, you have a reasonably good chance at recovery.

Getting the file back to the VMS system should be done using the same utilities but in the reverse direction. Then after the file is on the VMS system, you will need to either use convert if the file was originally transferred to the PC in ascii mode, or you will need to change the attributes of the file if the file was copied in image mode.

And as others have noted, FTPing indexed files when the purpose is for archival is not recommended.

I just tried doing the following, and verified that both worked.

I took a Prolog 3 indexed file with 30 keys and 60 areas (more than you likely had) that did have data (it was not an empty file). From a windows PC, I used the dos ftp client to transfer the file from the VMS system to the PC in ASCII (text) mode and IMAGE (bin) mode. Then transfered these files back to vms system to another directory.

The original indexed file was SCR:ITRC.ISM

C:\bin\> ftp vms_system_with_tcpip_5.4
ftp> asc
ftp> cd scr:
ftp> get itrc.ism itrc.asc
ftp> bin
ftp> get itrc.ism itrc.bin
ftp> cd itrc:
ftp> asc
ftp> put itrc.asc itrc.asc
ftp> bin
ftp> put itrc.bin itrc.bin
ftp> quit

In the case of the itrc.asc file, it is a sequential file that has all the data, but none of the key structure. The itrc.bin file is exactly the same in the contents of the blocks used by the file, but the file header does not have the correct file attributes.

This can be verified with the differences and backup/compare.

$ differences scr:itrc.ism itrc:itrc.asc
Number of difference sections found: 0
Number of difference records found: 0
$ backup/compare scr:itrc.ism itrc:itrc.bin
$ sho sym $status
$STATUS == "%X10000001"
$

Now you need to recreate the part of the file that was lost during the transfer, and for that you will need a file with the same structure as the file originally had. From that we can get the file attributes for the binary file, or an FDL we can use to convert the sequential file into an indexed file.

In the experiment I did, I used Joe Meadow's FILE program to fix the .bin file and analyze/rms/fdl/out=itrc.ana to create an fdl file.

$ set def itrc:
$ file scr:itrc.ism /out=itrc.fix
$ edit itrc.fix ! change itrc.ism to itrc.bin
$ @itrc.fix ! this applies file attributes from scr:itrc.ism to itrc:itrc.bin
$ anal/rms itrc.bin
...

The analysis uncovered NO errors.

$

Recover from asc mode transfer:

$ anal/rms/fdl/out=itrc.ana scr:itrc.ism
$ convert/fdl=itrc.ana itrc:itrc.asc itrc:itrc.new

At this point both itrc.bin and itrc.new should be logically equivalent to the original file. At least they should be usable.


NOTE WELL: If the file was transferred to the PC with ASC mode, it must use ASC mode on the return trip. It the file was transferred to the PC with BIN mode, you must use BIN mode to transfer it back. Hopefully whatever transferred to the "archival" system preserved everything as it was on the PC, otherwise all bets are off.

Good Luck,

Jon

[Steven Schweda]

> Backup of the file to a saveset followed by
> zip "-V" [...]

If you insist on wearing a belt _and_
suspenders.

> [...] whether you want to save multiple
> versions [...]

"zip -w" As "zip -h" says:

-w append version number to stored name

> Zip 2.32 works for every indexed file I
> have tossed at it, [...]

What could go wrong? (And be sure to
complain when it does.)