topic Re: Is there a utility to open and display/copy a file via FILE ID? in Operating System - OpenVMS

Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Tue, 10 Jun 2008 20:42:37 GMT

Before I reinvent the wheel... I would basically like:

copy/id=(x,y,z) device: [outputfile]

The undocumented dump/id would be usable, but doesn't really seem to open a file by ID, it seems to just look up that file header to get the filename and then open it normally, so I think it doesn't work for files without directory entries.

I haven't found anything out there so far...

Re: Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Tue, 10 Jun 2008 21:07:49 GMT

Some background, I'm asking because spooled print files get created on the intermediate disk device without directory entries. I can get their file IDs using getqui. I'd like to write a procedure that can display or copy the files, given a queue entry number.

This would be much better than the current alternative of running an/dis/rep, and then poking around for a file that's the same size/owner/date of the relevant entry.

Re: Is there a utility to open and display/copy a file via FILE ID?

Jon Pinkley — Tue, 10 Jun 2008 23:22:08 GMT

I think you used to be able to enter a name into a file directory using PIP. But that doesn't help here.

Joe Meadow's FILE program allows you to do this: From the FILE.HLP file

/FID

You may use this qualifier to specify the file id of a
file to enter, with the name passed on the command line. For
instance:
$ FILE GARBAGE.SPOOLED/FID=(1234,56,0)

This will enter the file specified by the file id
(1234,56,0) into your current directory with a name of
GARBAGE.SPOOLED. You may at this point do other things to the
file (such as make a backup copy).

The original intent behind this was to be able to recover
temporary spool files (which don't get created in any
directory, and are created "delete on close", thus having the
"marked_for_delete" flag set).

Here's a something Hein wrote a while ago:

http://mvb.saic.com/freeware/vmslt04b/vu/type-file-by-fid-vms.txt

Jon

Re: Is there a utility to open and display/copy a file via FILE ID?

David B Sneddon — Wed, 11 Jun 2008 00:59:14 GMT

Tom,

I have a patch_file utility (available on the freeware CDs in
the DBS-PATCH package) that can open a file by file id.

The following example uses a file that IS in a directory
but since you will know the device and file id...

tardis_FTA7> dir /file zap.obj

Directory DBS0:[SCRATCH]

ZAP.OBJ;1 (20967,9,0)

Total of 1 file.

tardis_FTA7> pfile dev dsa6000: id 20967 9 0
Patch_File V2.0
Patching file DISK$DALEK:[SYSPROG.DSNEDDON.SCRATCH]ZAP.OBJ;1
DSA6000: (20967,9,0)

PatchFile set disp ascii read 1 disp
Reading VBN 1 (00000001)

Modified VBN 1 (00000001)
0-[@...@.............Ã¸.....MAIN..V113-DEC-2006 01:17...............]
64-[..........AMAC V5.0-120......................... ABS .........]
128-[Ã ....... BLANK ................HELP.............Ã .<....CODE.....]
192-[...............DATA............. ......$ABS$...................$]
256-[LINKAGE.....(..............................BUF......(...J.p.....]
320-[...................START..Ã ...Ã .....................=...........]
384-[....................4...=..........## read VBN ##......X fo]
448-[llowed by hexadecimal number to read VBN ##....S followed by ]

PatchFile Exit

The source is available and it should be easy to add a copy function.

Dave

Re: Is there a utility to open and display/copy a file via FILE ID?

Hoff — Wed, 11 Jun 2008 00:59:50 GMT

What might arising here that causes a need to reference files by FID? Some background, please?

I'm guessing at some possibilities here...

There's a bug that causes spooled entries to get stuffed up and left in in the file structure. That was fixed a while back (XQP?), and patches are available for various versions.

There are queue symbionts that allow you to process files available via Freeware or such. Here's one, and it can serve as a Google target for finding others.

http://mvb.saic.com/freeware/freewarev80/execsymb/

Re: Is there a utility to open and display/copy a file via FILE ID?

John Gillings — Wed, 11 Jun 2008 01:08:40 GMT

Tom,

If you're running V8.2 or higher, it might be interesting to see what F$FID_TO_NAME returns.

>then poking around for a file that's the
>same size/owner/date of the relevant entry.

If you "repair" the file into SYSLOST, you can positively identify it by comparing F$FILE(f,"FID") with your known FID.

If you don't want to use ANALYZE/DISK/REPAIR, it's fairly simple to create a directory entry for it using an ACP $QIO (see I/O Users Guide).

Re: Is there a utility to open and display/copy a file via FILE ID?

Hein van den Heuvel — Wed, 11 Jun 2008 02:37:47 GMT

Just ENTER the file into a directory.
Then manipulate / copy / display the file to your hearts content.
Attached is a trivial tool to do just that.
It is much similar to the type tool Jon already posted a link to.

hth,
Hein

Re: Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Wed, 11 Jun 2008 14:52:24 GMT

Thanks for the responses. I KNEW there were utilities out there already, I just couldn't remember/find them...

FILE and VERB are a couple of the most useful add-ons ever...

Re: Is there a utility to open and display/copy a file via FILE ID?

Derek Garson — Wed, 25 Jun 2008 01:24:54 GMT

> If you "repair" the file into SYSLOST, you can positively identify it by comparing F$FILE(f,"FID") with your known FID.

If the file exists in any known directory then you can use FID syntax to access it e.g.

$ type DDcu:[syslost]~[x,y,z]

Unfortunately that doesn't seem to work unless the file exists in a directory and you know the directory path.

Re: Is there a utility to open and display/copy a file via FILE ID?

Hein van den Heuvel — Wed, 25 Jun 2008 02:27:46 GMT

In case folks are wondering, Derek is refering to the little known "FID-Abbreviated Names" as documented (somewhat) in:

http://h71000.www7.hp.com/doc/731FINAL/4506/4506pro_019.html#fid_abbrev_names

These are only valid on ODS5 disks,
swith process parse style set to extended.

Notice how they are call abbreviated NAMES, not specification.
It refers only to the name part of a name. RMS still does a Directory lookup, but it hunts for the FID not just a name.

Derek, and myself, feel this is a missed opportunity to allow for FID-only file specifications in general. I suspect this was not carried through as parse output gets difficult/partial. I would happily accept that restriction (gigo), but it was deemed too different from the norm.

Sample usage session below.
There are also DID-Abbreviated Directories

Hein.

$ set def myods5
$ dir
%DIRECT-W-NOFILES, no files found
$ crea aap.txt
:
$ dir/fil
Directory USR_ODS5:[HEIN]
aap.txt;1 (213,1,0)
:
$ type USR_ODS5:[HEIN]~[213,1,0]
:
$ type USR_ODS5:[xxxx]~[213,1,0]
%TYPE-W-SEARCHFAIL, error searching for USR_ODS5:[XXXX]~[213,1,0].LIS;
-RMS-E-DNF, directory not found
-SYSTEM-W-NOSUCHFILE, no such file
$ dir USR_ODS5:[hein]~[213,1,0]
Directory USR_ODS5:[HEIN]
aap[213,1,0].txt;1
:
$ set def [test]
$ dir USR_ODS5:~[213,1,0]
%DIRECT-E-OPENIN, error opening USR_ODS5:[TEST]~[213,1,0].*;* as input
-RMS-E-DNF, directory not found
-SYSTEM-W-NOSUCHFILE, no such file

Re: Is there a utility to open and display/copy a file via FILE ID?

John Gillings — Wed, 25 Jun 2008 03:46:28 GMT

Aha ODS-5 - that's why I couldn't make it work! I didn't think Derek would give us a bum steer ;-)

Re: Is there a utility to open and display/copy a file via FILE ID?

Derek Garson — Wed, 25 Jun 2008 04:43:58 GMT

Right. I overlooked that that feature (FID abbreviation) is ODS-5-only, as it worked for me.

I guess that is for compatibility as much as anything else. I can't see any specific reason why the FID syntax doesn't make sense on ODS-2. It's just that you never need it because file specs can't be too long.

As far as the original problem goes I was really hoping that

DDcu:[0,0,0]~[x,y,z]

would do the trick. It should almost work since 0 in the DID field in a NAM/FIB would suppress the directory lookup and just allow access by FID. (-: But, no.

As Hein says, that would then require yet another special case with file spec defaulting.

Re: Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Wed, 25 Jun 2008 15:20:00 GMT

Thanks for the info Derek, I can't apply it to this specific case, but I was not aware of that little trick, you learn something every day. Also learned you can reopen a thread.

As a followup, I'm using the example code Hein posted, which is perfect for my application. Thank you!!

Re: Is there a utility to open and display/copy a file via FILE ID?

Jess Goodman — Wed, 25 Jun 2008 15:29:46 GMT

Another good reason that FID-only file specifications are not allowed for any RMS file reference is that it would bypass any permission restrictions on the directory. Even if this was only allowed with a privilege such as BYPASS it would still skip any security auditing set on the directory.

However I just did a test on my VMS 7.3-2 (fully patched) system and was surprised to see that you can skip the directory protection check on any outer directories by using a DID-abbreviation to point directly to an unprotected sub-directory.

$ DIRE/PROT/FI DISK$SAN016:[000000]ODS5TEST,[ODS5TEST]sub

Directory DISK$SAN016:[000000]

ODS5TEST.DIR;1 (2547,1,0) (RWED,RWE,RWE,)

Total of 1 file.

Directory DISK$SAN016:[ODS5TEST]

sub.DIR;1 (2620,7,0) (RWE,RWE,RWE,RE)

Total of 1 file.

Grand total of 2 directories, 2 files.
$ SET PROCESS/PRIV=NOALL

$ COPY DISK$SAN016:[ODS5TEST.sub]JUNK.TXT;1 NL:/LOG
%COPY-E-OPENIN, error opening DISK$SAN016:[ODS5TEST.SUB]JUNK.TXT;1 as input
-RMS-E-PRV, insufficient privilege or file protection violation

$ COPY DISK$SAN016:[2620,7,0]JUNK.TXT NL:/LOG
%COPY-S-COPIED, DISK$SAN016:[2620,7,0]JUNK.TXT;1 copied to NL: (1 record)

Other commands that access the file this way seem to work too, with a notable exception:
$ TYPE DISK$SAN016:[2620,7,0]JUNK.TXT
%TYPE-F-SYSERROR, system service error
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation

But the strangest result from my tests is that using the DID-abreviation path to get to JUNK.TXT works with the DUMP command, but using JUNK.TXT's file ID with DUMP's /ID= qualifer does not work. I think this is because TYPE and DUMP /ID both use the back link file identification to get the full path to the file.

Re: Is there a utility to open and display/copy a file via FILE ID?

Hein van den Heuvel — Wed, 25 Jun 2008 15:45:42 GMT

>> Another good reason that FID-only file specifications are not allowed for any RMS file reference is that it would bypass any permission restrictions on the directory.

I beg to differ.

All a directory does is to provide a name to FID translation. If you _know_ the FID, then clearly the directory protection is irrelevant.

The object (file) protection is the only thing that really matters. Directory protection is just a minor 'hurdle' / inconvenience to stop simple undesired access attempts.

One could conceivably just brutefoce your way through all ID/SEQ combination and nothing would stop you. Only resource consumption would conspiciously high.

You could argue that when you know the FID, the directory protection should allow/prohit you from back-translating to a name, which it is indeed doing.

>> Even if this was only allowed with a privilege such as BYPASS it would still skip any security auditing set on the directory.

bah humbug. Only the object protection reallly matter. SYS$OPEN and SYS$QIO have full support for open by file ID.

>> $ TYPE DISK$SAN016:[2620,7,0]JUNK.TXT
%TYPE-F-SYSERROR, system service error
-SYSTEM-F-NOPRIV, insufficient privilege or object protection violation

Hmmm intersting.

>> DUMP's /ID= qualifer does not work. I think this is because TYPE and DUMP /ID both use the back link file identification to get the full path to the file.

Yeah... and IMHO that is a big and annoying bug. The LIB$FID_TO_NAME call used by DUMP should be use in an advisory and thus optional role. It should not break the main function as it does today.

Cheers,
Hein.

Re: Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Wed, 25 Jun 2008 15:55:02 GMT

Yeah, the undocumented dump/id was one of the first things I tried, but it dosn't do what I had hoped at all.

Besides, after entering a filename, I can do anything, not just dump it.

Re: Is there a utility to open and display/copy a file via FILE ID?

Tom O'Toole — Wed, 25 Jun 2008 15:56:23 GMT

Of course, if it's a choice between fixing a bug in an undocumented function of dump, and increasing the maximum tape blocksize, you know where my vote goes.

Re: Is there a utility to open and display/copy a file via FILE ID?

Jess Goodman — Wed, 25 Jun 2008 16:09:45 GMT

Actually the DUMP /ID= feature has been documented since at least VMS 7.2.

Re: Is there a utility to open and display/copy a file via FILE ID?

Jon Pinkley — Wed, 25 Jun 2008 17:13:24 GMT

Here's a summarized demo showing that directory protection is only protecting the file names/file id (and to an extent, the ability to delete a file, for example, I wasn't able to delete the file using DFU's delete by file/id. See attachment for more complete example.

You can easilty replace the contents of the file if you have write access to it, regardless of the protection of the primary directory the entry is in.

OT$ cre /dir [.protected]
OT$ set default [.protected]
OT$ cre file_with_world_access.txt
This file is in a protected directory, but the file has w:rwed
Exit
OT$ set prot=w:rwed FILE_WITH_WORLD_ACCESS.TXT;1
OT$ set def [-]
OT$ set prot=(g,w) protected.dir;1

From non-prived account:

TSTO:[PINKLEY_1]> type ROOT$USERS:[JON.protected]FILE_WITH_WORLD_ACCESS.TXT;1
%TYPE-W-SEARCHFAIL, error searching for ROOT$USERS:[JON.PROTECTED]FILE_WITH_WORLD_ACCESS.TXT;1
-RMS-E-PRV, insufficient privilege or file protection violation
TSTO:[PINKLEY_1]> cre /dir [.play]
TSTO:[PINKLEY_1]> set def [.play]
TSTO:[PINKLEY_1]> file jonsfile.txt/fid=(58964,79,0)
%SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
TSTO:[PINKLEY_1]> type jonsfile.txt
This file is in a protected directory, but the file has w:rwed
TSTO:[PINKLEY_1]> type jonsfile.txt
This file is in a protected directory, but the file has w:rwed
TSTO:[PINKLEY_1]> crea new.txt
This will replace contents of original file.
Exit
TSTO:[PINKLEY_1]> copy/over new.txt jonsfile.txt
%COPY-I-OVERLAY, ROOT$USERS:[PINKLEY.PLAY]JONSFILE.TXT;1 being overwritten
%COPY-S-COPIED, ROOT$USERS:[PINKLEY.PLAY]NEW.TXT;1 copied to ROOT$USERS:[PINKLEY.PLAY]JONSFILE.TXT;1 (1 block)
TSTO:[PINKLEY_1]> type jonsfile.txt
This will replace contents of original file.
TSTO:[PINKLEY_1]> log
PINKLEY logged out at 25-JUN-2008 13:51:53.53
%HGLOGIN-I-EXIT, returning to original JON session
OT$ type [.protected]FILE_WITH_WORLD_ACCESS.TXT;1
This will replace contents of original file.
OT$

Re: Is there a utility to open and display/copy a file via FILE ID?

John Gillings — Wed, 25 Jun 2008 21:03:14 GMT

Tom,

If you want to avoid making a directory entry, and what you're attempting to recover is text, and you have heavy privileges, you can use DUMP, but it's rather round about... (a procedure to automate this is feasible, but it involves a lot of parsing).

In this example I have a file called "JUNK.TXT" with FID (16193,2,0) on physical device $1$DGA6298:

First (SYSPRV or BYPASS required) access the header of the file using DUMP. This requires the "magic" INDEXF.SYS FID offset - this is used to calculate the block in INDEXF.SYS containing the header. There's a formula based on maximum files allowed and cluster size, but for one-off use, the simplest method is to dump the block of your FID, see what FID is returned and use the difference as your offset. In my case

$ dump/file/block=(s:16193,c:1) $1$DGA6298:[000000]indexf.sys

gives me:
File identification: (15907,3,0)

So calculate my target block as:

16193*2-15907 = 16479

$ dump/file/block=(s:16479,c:1) $1$DGA6298:[000000]indexf.sys

Sanity check the FID, backlink and name.

File identification: (16193,2,0)
...
Back link file identification: (0,0,0)
...
Identification area
File name: JUNK.TXT;1

FID is correct, backlink is (0,0,0), meaning "lost" and filename is as I expect.

Now look at the map area:

Map area
Retrieval pointers
Count: 41 LBN: 9430

With LOG_IO privilege, the extents can be dumped by dumping LBNs from the device. I won't dump out the whole cluster, since I know there's only one record in the file.

$ dump/block=(s:9430,c:1) $1$DGA6298:
Dump of device $1$DGA6298: on 26-JUN-2008 07:57:59.79

Logical block number 9430 (000024D6), 512 (0200) bytes

00000000 0000FFFF 00747874 2E6B6E75 6A206E69 20747865 7420656D 6F730015 ..some text in junk.txt......... 000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000020
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000040
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000060
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000080
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0000A0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0000C0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0000E0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000100
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000120
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000140
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000160
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 000180
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0001A0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0001C0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ................................ 0001E0
$

Obviously this will get complex if there are many extents and/or extension headers. You also need to interpret the dumped output to reconstruct records.

However, if all you want to do is visually inspect the contents of the file, it might avoid the need to write a program to create a directory entry.