Confused about error from CONVERT/FDL

[Mike Myers_6]

Last week I received the following from a weekly job running CONVERT/FDL

%CONV-F-OPENOUT, error opening DSA1:[SMV.DISK2]MFG_INSTRUCTIONS.DAT;705 as output
-RMS-E-FLK, file currently locked by another user

My confusion comes due to the fact that the CONVERT/FDL step is actually creating this file. (Version 704 was input, 705 is new.) How could any other user actually have this file locked when it is just now being created by this command?

Is it possible that the CONVERT command in fact momentarily closes the file after building the primary index and opens it again?

Am I just confused by the error message that says "as output"? Could this mean "to update"?

Seems like convert should have kept the file open & locked until is completed processing.

We have been running this job for many years and this is the first I can recall this happening. This is on OpenVMS V7.1-2.

Any ideas beyond the obvious? I'd like to avoid a repeat.

[Martin P.J. Zinser]

Hello Mike,

I am guessing here, Hein might have more details. Essentially I can think about two ways convert/fdl could work.

1.) Create the indexed output file, then populate this file with the records from the input file. It should be possible to do this in an "atomic" way.

2.) Extract the records from the input file to populate a sequential file. Then create the index structure for the new file. It seems that there might be a chance in this type of process to run into an error like the one you observed.

Since the performance of variant 1 is not very good and observation shows that convert/fdl is actually pretty fast, I suppose something along the lines of variant two is actually imprlemented.

The chances of actually experiencing this problem are pretty slim (also requires that a second application tries to get at the file while you do the convert).

One way to prevent this from happening might be to do a convert/fdl orig temp and then a
rename temp orig

Greetings, Martin

[Hein van den Heuvel · Accepted solution]

[ Hi Martin! Nice to have met you last week]

Mike> Any ideas beyond the obvious? I'd like to avoid a repeat.... ((Version 704 was input, 705 is new.)

How about the obvious: Do not convert to a target name that corresponds with the production file name if there is the slightest chance that a production job could touch the file.

You don't want your production tasks to try to open a partly converted file no? Convert, then rename!

Yes, convert _could_ use a temp name, or it could use no name (file ID only) and only enter a name when done, but that's just not how it is done. This is mostly not done because you'd be pretty upset if after 5 hours converting there was a problem with creating the target name!

Mike> This is on OpenVMS V7.1-2.

For the version and earlier the FAST_LOAD option for the convert utility builds a new indexed file by building one index at a time.
After the build of the primary index, it closes the output file and passes it to the sort utility for a sort of the first secondary key.
When the sort utility returns control to convert, it reopens the output file to build the index for the first secondary key. This same process is followed for any additional
secondary key.

Some other process could gain control of the file during the window when sort closes the file and convert attempts to reopen it.

VMS 7.2 and better changed convert to use the record interface to calleable sort, so convert can keep the file open (best I recall!). See release notes!


Martin> Since the performance of variant 1 is not very good and observation shows that convert/fdl is actually pretty fast, I suppose something along the lines of variant two is actually imprlemented.

Actually... convert does use the data records directly from the input file.
In some rare occasions it is faster to roll out to sequential. You can witness this when converting bucket sizes. Say the input was 1000 buckets of 2 blocks each. Make the output 100 buckets of 20 blocks (plus one index bucket). Convert old to new will take 1101 IOs (give or take :-). 1000 reads, 100 data writes, 1 index write. Convert new to newer and you'll see 201 IOs.
Convert is fast because
- It double buffers output, performing an Async IO on a full buffer while buildnig the next
- It only writes out full index buckets.
- It understands SORT.

Does that help?
Cheers,
Hein.

[Mike Myers_6]

Hein,

You provided the insider information I was looking for:
Hein>For the version and earlier the FAST_LOAD option for the convert utility builds a new indexed file by building one index at a time.
After the build of the primary index, it closes the output file and passes it to the sort utility for a sort of the first secondary key.
When the sort utility returns control to convert, it reopens the output file to build the index for the first secondary key. This same process is followed for any additional
secondary key.

Some other process could gain control of the file during the window when sort closes the file and convert attempts to reopen it.

VMS 7.2 and better changed convert to use the record interface to calleable sort, so convert can keep the file open (best I recall!). See release notes!


I suspected something like this was the case and had reason to believe that the keys were built in stages. I just was not sure that CONVERT actually closed the file at any time.

This application is supposed to tbe *quiet* when this is running. No users or other batch processes. If users are on they would likely be updating this file and we could not convert it anyway. (operator screw up somewhere) The output to another name and rename idea would not be appropriate unless we could be assured that all writers were out of the file. I believe this system is one of the few left to be upgraded to 7.2 soon ;) anyway.

Thanks for your assistance,
Mike

[Stanley F Quayle]

Instead of CONVERT and rename, how about rename and then CONVERT? Rename the original file to a temporary name.

If everyone's supposed to be out of the file, renaming it should be completely safe -- if someone messes up, they'll get a "file not found" error. Serves them right.

Then, after CONVERT is done with the file (to a different temporary name), rename the output file to the usual name.