Why do directories need to be contiguous ?

[Wim Van den Wyngaert]

Wim

[Hein van den Heuvel]

I'll try to dig this one up (ask Andy if/wehn I run into him :-)

three thoughts

- It's been that way for ever, noone dares touching it.

- Directories are used during the bootstrap, before the xqp is active to do the window turns, so all directories in the boot path need to be contiguous.

- Avoid that split IO for a relatively small file which is relatively frequently read in general. (Yes some applications have large directories which are rarely read, but in general...).


Oddly enough, counter to the last points, for the first 20 years of VMS directories were always written a single block at a time. Again,,, in general VMS _expects _ to read them much more often then write them.

Did you check the 'VMS filesystem internal black book'?

Maybe more later...

Hein.

[Wim Van den Wyngaert]

Thx Hein. But don't have that book (only internals 5.5).

Wim

[Robert Gezelter]

Wim,

I do have that book, just it is not with at this instant. I will try to take a look when I get a chance.

My recollection (dimly, from the distant past) was that the restriction was a function of the drivers used during the bootstrap process. If that was the case, I would probably be in favor of relaxing the restiction somewhat, at least on non-system residence volumes.

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Wim,

Universal answer "because". It was a design decision, certainly valid at the time given the state and cost of technology.

Is this curiosity or is it causing a real problem? I'd argue that if you're experiencing difficulties finding enough contiguous space for a directory, then either your volumes are WAY too fragmented, or you're trying to stuff too many files into the directory, or both.

The effort and risks involved in relaxing this restriction would far outweigh any benefits. I'd much prefer to see engineering effort put into developing a new file system that breaks the 1TB limit than expended on risky modifications like this (who knows where the assumptions have been made?).

If you're having trouble with directories, have a look at the DFU DIRECTORY command.

DIRECTORY/CREATE/ALLOCATE=n

preallocates space before the disk gets fragmented. Make it big enough and you'll never have to extend it.

DIRECTORY/COMPRESS/FILL_FACTOR

will compact down the buckets in an existing directory file, "defragging" it. If you fail to extend a directory file because there is insufficient contiguous space, try /COMPRESS. It may recover enough internal space to allow more files to be inserted.

With disk space costing about 10MB for 1c, it's easier, safer and (MUCH!) cheaper to solve any problems by adding more resource. A 1MB directory is about as big as they should ever get.

[Hein van den Heuvel]

I added a fourth thought in an Email to Andy G.

Hein> - It allows for special casing in the XQP code avoiding window control blocks for directories.

His answer:
---------------------------------------------------------------------------
Bingo on the last point. The file system handles directories without
WCBs - using only FCBs made it more palatable to cache the access info
in the early days when memory was tight. That was the original rationale.

In retrospect, it also makes the boot path simpler, although I'm not
sure by how much, since the boot file system can handle non-contiguous
files (up to a limit).
------------------------------------------------

Hein.

[John Gillings]

Another reason why relaxing the restriction is very non-trivial - rolling upgrades and mixed version clusters.

Suppose non-contiguous directories were implemented in (say) V8.4. We'd need some attribute on the volume to say they were allowed. Now any pre V8.4 systems could not mount the volume, because they don't understand how to read non-contiguous directories. But those systems don't know about the feature, so how to stop them mounting the volumes? Back port patches? More work for engineering, taking resources away from more important stuff.

Sure that's true today for a few cases - extended bitmaps and ODS-5, but in those cases the change has greatly extended the capabilities of the volumes, so the pain is worthwhile. The question arises, who would (really) benefit from any proposed change to contiguity of directory files? Examples welcome.

Indeed it occurs to me that a change that might be more worthwhile would be to create additional contiguity rules in the file system. We have "best try contiguous" files, but, in reality it doesn't try very hard at all. I'd like to see a "try very very hard contiguous", or even a mechanism to say that all (say) .EXE files should be contiguous. Since they don't get extended, it should be possible to keep them contiguous at all times.

With larger and larger volumes of cheap storage, some of the reasons for allowing fragmented files are much less compelling than they were when they were created. Design decisions need to be reevaluated in the light of changes in technology and economics.

[David Jones_21]

"Indeed it occurs to me that a change that might be more worthwhile would be to create additional contiguity rules in the file system. We have "best try contiguous" files, but, in reality it doesn't try very hard at all. I'd like to see a "try very very hard contiguous", or even a mechanism to say that all (say) .EXE files should be contiguous. Since they don't get extended, it should be possible to keep them contiguous at all times."

How about replacing bitmap.sys with something not organized around a single simple bitmap? You could break up the disk space into different zones, for example, with different policies for different zones. Files could have an additional attribute indicating the zone they use, by default inheriting the zone of the parent directory.

[Robert Gezelter]

John,

Actually, I would settle for a /PREEXTEND switch on the CREATE/DIRECTORY command (and have stated this opinion in the past). This would allow directories to be pre-extended, which would, at least in my experience, eliminate the overwhelming percentage of the cases in real operations.

The fact that DFU can do it is besides the point. Many sites would find the option on CREATE/DIRECTORY and be aware of it, relative to the number who figure out the existence of DFU.

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

Bob, this exists....

DCL DICTIONARY 8.2
CREATE/DIRECTORY
:
Qualifiers
/ALLOCATION=n Specifies the initial number of blocks to be allocated to each of the specified directories. The default allocation is 1 block.

Too bad Wim never told us WHY he wanted to know. I suspected it was just curiosity, until the answer with the solution confirmed by Andy was not marked as such.

John,

I like those suggestions
- no real need for non-contuguous directories
- re-born aggresive contiguous files without WCB (window control block) needed for the FCB
- contiguous best try to be made more aggresive and never dropped as attribute.


David,
Some zoning would be neat, but how to provide the semantics? Not too many bits left in fib and fabs and the likes.
I recommend considering partitions (HSG/EVA/LD) to achieve this for now.

For files that are expected to live for a longer time it is a shame to have them be fragmented, and have them fragment freespace. For temp files, daily logs, it is a waste of time to worry about defragging, other then the effect on others. Maybe an occasional defrag is good enough to solve this, notably if the defrag tool were to take the file age into account. Does any defragger do that? Wouldn't it be useful to say 'defrag files between N and M days old?

cheers,
Hein.