Return to the top of a sequential file

[Jesse Chambless]

Hi all,

Could someone please assist me in "rewinding" the file pointer to the top of a file in DCL?

I have 2 files.

I open file 1 and read in each line.

I then open file 2.

I need to read through file 2 until I find a specific match.

I then need to read the next line in file 1, and start at the top of file 2 to begin a new search.

I have not been able to find a way to reset the file pointer to the top of file 2.

I know I could close file 2 and then re-open it thus resetting the pointer.

Thanks in advance.

[Thomas Ritter]

If you must use DCL then use must close and reopen the second file.

If the file is large say 100,000 records try the same process but using C or Cobol. The runtimes and impact on the system will be must less.

Over the years lots of DCL performing file I/O work has been replaced with C. In C you can read backwards and forwards. Very useful for some times of pattern matching exercises.

[Willem Grooters]

Depending on what your files look like, and what you try to achive, it might be an idea to use SEARCH in the second file if you can specify exactly what you need to find. Then you won't need to open file 2 for this. Otherwise, you will need to close and open the file time after time again.

[Andrew G Commons]

Sounds like you are almost doing a merge of the two files. Have you considered sorting the two files into the same order and doing your 'merge' in one pass through both files. You can use sort specification files to specify arbitrary keys and collating sequences so it might be possible to get the two files in the same order from the perspective of the keys you are matching on.

cheers
andrew

[Robert_Boyd]

If you can construct file 2 so that it is indexed (even if the index is just a record number) then you can reset the pointer by using the key to go back to the 1st record.

I agree with the others though, that if the file is very large you would be better off to use perl or c to do this kind of project to get the greater efficiencies of these languages.

Robert

[Robert Gezelter]

Jesse,

Presuming that you want to read the entire file twice, the easiest way to do this would be to close the file and then re-open it.

If you want to do slightly more subtle work, you can use the RMS calls to note the position of the file at the beginning, and re-position the file to the beginning (see the RMS User's Guide for details).

Since your description could involve many passes, I would definitely use the ability to note the position of the file, and resposition it, rather than opening it again and again. I would also check the use of readahead and multiple buffers and extended blocksizes.

- Bob Gezelter, http://www.rlgsc.com

[Tony_289]

Hi Jesse,

I'll offer another methodology. I have used TPU (and LSE) within DCL command files to scour through data files. Similar to Thomas' C reply, one can go forwards, backwards and search within TPU, but it depends on how efficient you would want to be and/or what you are trying to accomplish.

Are you bounded within DCL? If so, then I would close and re-open the 2nd file as needed.

Regards,

Tony Flores

[Hein van den Heuvel]

>> Could someone please assist me in "rewinding" the file pointer to the top of a file in DCL.

No can do. This is a semi-documented RMS restriction. VMS Guide to file applicaitions, Process permanent files:
http://h71000.www7.hp.com/doc/731FINAL/4506/4506pro_019.html#apps_using_proc_permanent_file

Any attempt to do so, will a
%RMS-F-IOP, operation invalid for file organization or device

allthough... that is only supposed to happen for SQO opened filed.
If you could do this, then you could write a small program to connect to the already opened file and rewind it.

Anyway, by the time you started an image, you've wasted more then you wanted to save.
So just CLOSE the file and re-OPEN.


Mind you, I agree with the others that the whole notion of re-scanning a second file is impractivally slow, and thoroughly in-elegant for anything but the smallest files.

Please describe the contents and actions more clearly, and we may be able to come up with a better perl, awk, dcl, c-program solution. You may want to use a temporary indexed file for larger datasets. For smaller datasets you may want to just remember a buch of data in dcl symbols.
And perl assosiative arrays are rather nicel solution for multi file cross reference work.

If you do explain more, be sure to provide a few lines of sample input data in a text attachement here, as the main forum text will mess up spaces and tabs.

hth,
Hein.

[Heinz W Genhart]

Hi Jesse

If the second file is not large, you could read the entire file line per line into variables.

$ idx=1
$ open/read in file.ext
$read_loop:
$ read/end_of_file=end_loop in line'idx'
$ idx = idx + 1
$ goto loop
$end_loop:
$ close in


You can then 'jump' back by using the variables line1 ... line n

If the file is large, I would recommend also to use C, to solve this problem

Regards

Heinz

[Jan van den Ende]

Jesse,

depending on HOW OFTEN you need to reset that file, you could consider OPENing it /SHARE (=READ maybe?).
Now you can open another logical name for the same file.
The number of times you can do that is limited.
One potential advantage: your pointer into the first channel is preserved, and so can be used again. (no idea of course if you have any use for that)

Another EUR 0.02

Proost.

Have one on me.

jpe

[Jesse Chambless]

Thanks for all the responses. I am not skilled in C so I'm stuck with DCL.

Here is what is really happening.

I have a CSV file of users that have been deleted for the application. It contains their Name and Account information but not the username. This contains approximately 3000 users.

I have used a modified UAF_Check.com procedure to produce a file with the username, account and name. The sysuaf contains 7044 entries.

I need to read each line of the CSV file, extract the Account field and then try and match that to the output of the UAF_CHECK.COM procedure.

I will then take the username and delete the record from the sysuaf.

I realize that initially I am spending a lot of time trying to create this automated procedure, but it should save me much time later when I have to do this again.

[Volker Halle]

Jesse,

consider using SEARCH in the sysuaf listing. This is much faster than using DCL READs (remember: DCL is an interpreter and also has to read, decode and process your DCL commands every time in the loop). You will also benefit from the XFC cache.

You could SEARCH for the account name string, output the results to a file (or use PIPE) and then generate the REMOVE command for UAF.

I bet, you'll get another load of replies on how to best do this. Programming questions often have lots of (valid) answers...

Volker.

[Hein van den Heuvel]

a few thousand is not too many to hold in symbols.

Or just go one step further and convert that uaf_check output file to an indexed file.
Then as you look through the csv file, you can read the uaf records by key.
Take the right action based on that.

Example:
$ cre tmp.tmp ! Note, space align to 12 character fields.
aap test1 blah blah
noot test2 fwegwrtherhnerj
mies test3 qwefqwfqwf
$ cre tmp.fdl
file; organization indexed; key 0; seg0_p 12; seg0_l 12
$ convert/stat/sort/fdl=tmp tmp.tmp tmp.idx
:
$cre tmp.csv
test2,bbbbbbbbbbbbbb
test1,aaaaaaaaaaaaaaa
$
$cre tmp.com
$ open/read/error=ooops idx tmp.idx
$ open/read/error=ooops csv tmp.csv
$ loop:
$ read/end=done csv record
$ account = f$elem(0,",",record)
$ key = f$fao("!12AS",account)
$ read/key=&key idx uaf
$ write sys$output "uaf record to delete: ", uaf
$ goto loop
$done:
$close csv
$close idx
$exit
$oops:
$stat = $status
$write sys$output "Failed to open a file"
$
$exit 'stat
$
$@tmp
uaf record to delete: noot test2 fwegwrtherhnerj
uaf record to delete: aap test1 blah blah


Of course the $WRITE needs to become the delete.
And the forum presentation takes out the spaces from the input uaf file, so if you want to try my example before using the uaf_check output, then you need to space fill the fields to become 12 chars each.

If you had posted a few sample input lines from the uaf_check and csv files, then we coudl have helped more to the point, but I think you get the drift no?

note... I use F$FAO to space fill the key.
You might not need that for your usage, and DCL+RMS will do a 'generic' key match. For example a key of "aa" will find record "aap"


hth,
Hein.

[Jesse Chambless]

Per your request.

Here is a sample of the CSV file.

xxxxxxxxxxxxxxxx,AAC,
xxxxxxxxxxxxxxxx,AAH,
xxxxxxxxxxxxxxxx,AAK,
xxxxxxxxxxxxxxxx,AAL,
xxxxxxxxxxxxxxxx,AAM,
xxxxxxxxxxxxxxxx,AAP,
xxxxxxxxxxxxxxxx,AAR,
xxxxxxxxxxxxxxxx,AAS,
xxxxxxxxxxxxxxxx,AAX,
xxxxxxxxxxxxxxxx,AAY,
xxxxxxxxxxxxxxxx,ABA,


Here is a sample of the uaf_check file.

AAA_CBO,AAA,xxxxxxxxxxxxxx
AAB_CCA,AAB,xxxxxxxxxxxxxx
AAD_LAB,AAD,xxxxxxxxxxxxxxx
AAE_MGP,AAE,xxxxxxxxxxxxxxx
AAF_AMC,AAF,xxxxxxxxxxxxxxx
AAG_MGP,AAG,xxxxxxxxxxxxxxx
AAI_ORS,AAI,xxxxxxxxxxxxxxx
AAJ_ORS,AAJ,xxxxxxxxxxxxxxx
AAK_AMC,AAK,xxxxxxxxxxxxxxx
AAL_PTACT,AAL,xxxxxxxxxxxxx
AAM_RAD,AAM,xxxxxxxxxxxxxxx

These are some really GREAT suggestions!!

[Robert_Boyd]

Now that I've seen the format of the 2 files and your description of what you want to do with the data for verifying one file against the other, I reiterate my original suggestion -- convert file 2 to an indexed file, just as Hein has now also suggested.

Instead of searching through the file you can read directly to the record(s) of interest by using the key.

If it is possible more than one record might match on a given key, then you must take that into account and read all of the records from the 1st one that matches the key until you satisfy your search or fail when you find a different search key value.

Sometimes it's easier to get the best suggestions when more of the entire goal and description of the available inputs is laid out.

All I can say is it's just too bad there isn't a lexical F$GETUAI. If there was, then you wouldn't need file_2 at all.

Robert

[Hein van den Heuvel · Accepted solution]

So that's easy then...
To build on the dcl example I alread had, you first convert the cvs data to an indexed file:

$ typ uaf_helper.com
$open/read csv uaf.csv
$open/writ seq uaf.seq
$loop:
$read/end=done csv record
$write seq f$fao("!12AS!12AS!AS",F$ELE(0,",",record), -
F$ELEM(1,",",record),F$ELE(2,",",record))
$goto loop
$done:
$exit
$
$@UAF_HELPER
$ conv/stat/sort/fdl=tmp uaf.seq tmp.idx
$
$@TMP
uaf record to delete: AAK_AMC AAK xxxxxxxxxxxxxxx

You'll need to improve the TMP.COM to deal with 'record not found' using READ/ERR=...


Also, not that we see all the data, here is a perl solution to the same problem:


$ typ tmp.pl
open UAF,"open TMP, "while () {
chomp;
($user,$acc,$misc) = split (/,/);
$users{$acc} = $user;
$misc{$acc} = $misc;
$i++;
}
print "$i UAF records processed\n";

while () {
chomp;
($x, $acc, $y) = split (/,/);
if ($user = $users{$acc}) {
print "REMOVE $user ! for account $acc\n";
} else {
print "! No username found for $acc\n";
}
}
$
$
$ perl tmp.pl
12 UAF records processed
! No username found for AAC
! No username found for AAH
REMOVE AAK_AMC ! for account AAK
REMOVE AAL_PTACT ! for account AAL
REMOVE AAM_RAD ! for account AAM
! No username found for AAP
! No username found for AAR
! No username found for AAS
! No username found for AAX
! No username found for AAY
! No username found for ABA
$


cheers,
Hein.

[Doug Phillips]

Or modify the modified UAF_check.com to create an ISAM file instead of a CSV.

[Robert Gezelter]

Jesse,

Some comments from the Dark Ages, but eminently applicable in this case.

This can be done in two quick passes:
Sort the PULL list into order by the common key
Sort a listing produced by AUTHORIZE into the same order

It is now a simple matter to scan the two files in a single pass and identify any users who are in the PULL list. The SORT command is emminently usable from DCL, and for files of this size is very fast (in core).

For those classical scholars among us, this is the proforma way of merging updates into a sequential tape file in one pass and putting the output onto an output tape.

- Bob Gezelter, http://www.rlgsc.com

[Andrew G Commons]

Do you have to do this on VMS?? Given the data is in CSV format using something like MS Access would be trivial:

(1) Flick files to PC (or something that is visible from the MS world)
(2) Import files into seperate tables
(3) Define the account as a key in each table
(4) Do a join

Probably take all of 5 minutes, no coding required.

[Jesse Chambless]

Thanks to all.

I have implemented Heins suggestion and it works great.

Thanks to all for your valuable suggestions.

[Jesse Chambless]

Thanks I have used Hein's suggestion and it works great.

Thanks again to all for your valuable insight and assistance.

It is always great to work with such great people!

[Robert_Boyd]

Hein,

It occurred to me after looking at this again: since Jesse already has been modifying UAF_CHECK, it would be easy to cut out the CONVERT step by using CREATE/FDL inside the modified UAF_CHECK to create the output file as an indexed file to begin with. Then when the file is opened for writes the records will already be in the the right setup with a change in the output format to fit the key structure.

Robert

[Hein van den Heuvel]

>>> It occurred to me after looking at this again: since Jesse already has been modifying UAF_CHECK, it would be easy to cut out the CONVERT step by using CREATE/FDL inside the modified UAF_CHECK to create the output file as an indexed file to begin with.

Absolutely. And I _thought_ i had suggested that also, but do not see it now.

fwiw... When you do something liek that, then for some specific task you want to use an existing file if it exists, but be ready to create one on the fly of it does not exist. To solve that, i tend to use constructions like:

$file = "some-name.data" ! Name you not be hardcoded multiple times
:
$OPEN/READ/WRITE/SHARE=WRITE/ERROR=create_file file 'file
$next:
:
$create_file:
$CREATE/LOG/FDL=SYS$INPUT: 'file
file; organization indexed; key 0; seg0_length 8
$OPEN/READ/WRITE/ERROR=oops file 'file
$GOTO next
$

hth,
Hein.