Overheads of large .DIR files?

[Hein van den Heuvel]

7-seconds on 3m50s is less than 4%.
Minor environmental impacts like the physical location of the blocks might explain this.

From a practical perspective, it does not seem interesting. But I can appreciate that you want to get to the bottom of this, having dug this deep. So you'll have to quantify the those 7 seconds. Are they just elapsed time (block placement, cache effects), or is there something interesting at play.
It could easily be something silly like a longer command or comment in your test script cause a loop to span two IO buffers for a dcl script versus one.

If I wanted to understand that (But for now I do not :-), then I would test any and all of:

1) Measure which resources changes (IO, CPU, Time). See 'time.com' script below as a simple way to time a command.

2) Reproduce on a RAMdisk
$ mcr sysman io connect mda1:/driver=sys$mddriver /noadapter ...
$ init/size=123456 mda1 ram
$ moun/sys mda1 ram ram

3) Reprocude on a LD drive and use the LD TRACE option to explain each and every IO in both cases.

note 1)
The 128 block growth corresponds with the 100 blocks, round up to the (64 block) cluster size

note 2)
Older remark by John: "In other words the block split, but did so in the middle rather than according to where the new file entry would be located."
Correct. That's what I tried to explain earlier still with the line (with typos fixed) "With the block split, the new entry will be written to the low, or high, part of the new blocks as dictated by its sorting order."

Hein

------------------- time.com --------------
$! All times in Hundreds of a second.
$ old_krnltim = F$GETJPI ("","KRNLTIM")
$ old_exectim = F$GETJPI ("","EXECTIM")
$ old_suprtim = F$GETJPI ("","SUPRTIM")
$ old_usertim = F$GETJPI ("","USERTIM")
$ old_dirio = F$GETJPI ("","DIRIO")
$ old_bufio = F$GETJPI ("","BUFIO")
$ old_time = F$TIME()
$!
$! Run the test
$!
$ 'p1 'p2 'p3 'p4 'p5 'p6 'p7
$!
$! Save the end time
$!
$! end_cputim = F$GETJPI ("","CPUTIM")
$ end_krnltim = F$GETJPI ("","KRNLTIM")
$ end_exectim = F$GETJPI ("","EXECTIM")
$ end_suprtim = F$GETJPI ("","SUPRTIM")
$ end_usertim = F$GETJPI ("","USERTIM")
$ end_dirio = F$GETJPI ("","DIRIO")
$ end_bufio = F$GETJPI ("","BUFIO")
$ end_time = F$TIME()
$
$ elapsed = 100*(f$cvtime(end_time,,"SECONDOFYEAR") - f$cvtime(old_time,,"SECONDOFYEAR"))
$ elapsed = elapsed + f$cvtime(end_time,,"HUNDREDTH") - f$cvtime(old_time,,"HUNDREDTH")
$
$ WRITE sys$output -
f$fao ( "Dirio=!6UL Bufio=!6UL Kernel=!4UL RMS=!4UL DCL=!4UL User=!4UL Elapsed=!6UL ", -
end_dirio - old_dirio, end_bufio - old_bufio, end_krnltim - old_krnltim, -
end_exectim - old_exectim, end_suprtim - old_suprtim, end_usertim - old_usertim, elapsed )
$exit

[John McL]

Here's the results of using the above, in both cases wrapping a procedure that created 5000 files.

After CREATE/DIR (no allocation)

Start: 19-APR-2010 13:07:09
End: 19-APR-2010 13:11:00

Dirio= 12089 Bufio= 60009 Kernel=2371 RMS=1214 DCL= 108 User= 187 Elapsed= 23091


After CREATE/DIR/ALLOCATE=1000

Start: 19-APR-2010 13:27:02
End: 19-APR-2010 13:31:00

Dirio= 11875 Bufio= 60009 Kernel=2479 RMS=1238 DCL= 87 User= 146 Elapsed= 23784

[John Gillings]

John,

> so can anyone explain why this should
> take more time?

Pure guesswork... It may have to do with the way directory entries are distributed across the allocated space, and how they get shuffled around to make space for new entries. That in turn would be dependent on the sequence of entries.

You might be able to shed some light by single stepping through the insertions, using DUMP/DIRECTORY to observe the internal changes.

That you care about this rings alarm bells! If it's that critical, I'd be thinking there's a fundamental architectural issue which should be changed, rather than trying to optimise your way out.

A few alternatives. If you send your log output to a "log server" process, instead of to a file, you can package it up any way you want, or ship it off to another system. Stash the log records in fewer files, possibly compressed on the fly, then present them back to consumers via a web server with search facilities, etc...

This can be done with minimal impact on the source process. You may even find some off the shelf components to do most of the work for you. For example, your log server could reformat the messages as SYSLOG and use a standard server.

[John McL]

"That you care about this rings alarm bells!" What? If someone was interested to find out why, it would ring alarm bells with me.

It's been suggested (see early in this thread) that preallocating the directory file size will be faster because it won't need to be expanded, which makes sense. So why is it that when I test it - and I have done several times with very similar results - that what in theory should be faster is actually slower on a machine where I am the only user?

I've put together a set of recommendations and preallocation was included. It's easy enough to remove, but when someone tells me that preallocation should be faster and asks why I didn't include it, what logical explanation can I give them?

[John Gillings]

John,

Sorry, I'm not alarmed at your curiosity, rather, if variations in directory performance matter that much, that it affects your process flow/design, you're probably looking in the wrong direction.

I'm not completely convinced that preallocating directories will always be faster. Space allocation for a directory is still a rare event, and it's not slow enough to make a significant contribution to the total time.

Preallocation and extent size matter for files which may grow very large, but for a directory which only grows to a few hundred blocks it's just noise.

The only time preallocation is likely to really matter for directories is on a highly fragmented disk, where finding a single extent large enough for the ultimate size of the directory might be a problem.

What is the sequence of the directory entries? I'd expect very different performance from inserting in order, reverse order or random.

(later)... but then experiment says otherwise! Here are my results. This system has a cluster size of 208, so I guess that means the effect of allocations less significant than for smaller cluster sizes.

CREFILES.COM counts up or down to the limit, so inserting files in order or reverse order. An increment of "*" means "random".

create/dir/prot=(o:rwed) [.test]
TEST.DIR;1 1/208
@time @crefiles [.test] 5000
Dirio= 25892 Bufio= 80006 Kernel=1299 RMS= 255 DCL= 38 User= 37 Elapsed= 3507
TEST.DIR;1 264/416
@time del [.test]*.*;
Dirio= 18691 Bufio= 5011 Kernel= 96 RMS= 12 DCL= 0 User= 0 Elapsed= 1409
del test.dir;
TEST.DIR;1 1/416
(repeat)
@time @crefiles [.test] 5000
Dirio= 25660 Bufio= 80006 Kernel=1302 RMS= 287 DCL= 31 User= 41 Elapsed= 3483


create/dir/prot=(o:rwed)/alloc=1000 [.test]
TEST.DIR;1 1/1040
@time @crefiles [.test] 5000
Dirio= 25935 Bufio= 80006 Kernel=1268 RMS= 297 DCL= 34 User= 36 Elapsed= 3496
TEST.DIR;1 264/1040
@time del [.test]*.*;
Dirio= 18687 Bufio= 5011 Kernel= 100 RMS= 21 DCL= 0 User= 2 Elapsed= 1407
del test.dir;
(repeat)
@time @crefiles [.test] 5000
Dirio= 25936 Bufio= 80006 Kernel=1267 RMS= 291 DCL= 41 User= 25 Elapsed= 3505



create/dir/prot=(o:rwed) [.test]
@time @crefiles [.test] 0 -1 5000
Dirio= 37604 Bufio= 80006 Kernel=1272 RMS= 276 DCL= 33 User= 32 Elapsed= 4140
@time del [.test]*.*;
Dirio= 25787 Bufio= 5011 Kernel= 115 RMS= 8 DCL= 0 User= 1 Elapsed= 1826
del test.dir;



create/dir/prot=(o:rwed)/alloc=1000 [.test]
@time @crefiles [.test] 0 -1 5000
Dirio= 37290 Bufio= 80006 Kernel=1239 RMS= 293 DCL= 31 User= 35 Elapsed= 4102
TEST.DIR;1 500/1040
@time del [.test]*.*;
Dirio= 25818 Bufio= 5011 Kernel= 123 RMS= 25 DCL= 0 User= 0 Elapsed= 2965
TEST.DIR;1 1/1040

create/dir/prot=(o:rwed) [.test]
TEST.DIR;1 1/208
@time @crefiles [.test] 5000 *
Dirio= 34255 Bufio= 79990 Kernel=1341 RMS= 286 DCL= 74 User= 23 Elapsed= 3739
TEST.DIR;1 328/416

Note that my attempt at a random number generator in DCL didn't work very well, so there are some files with up to 5 versions, which has improved the stats (since adding a new version for an existing entry is faster than adding a new entry). My timing shows negligible difference with and without preallocation, but note that the used space for the directory is very different for the same set of files, demonstrating that inserting a directory entry is dependent on the directory size. For some insertion sequences, that will effect the chances of a "hole" where you want to insert, and therefore reduce the number of shuffles required.

[John McL]

I completely agree with you about file extensions and EOF movement for files in random order.

From the documentation that I'm putting together "After creating all the file entries in forward order, the directory file size was 123 blocks with 128 block allocation. Creating files in reverse order used 249 blocks (320 blocks allocated) due to extra block splitting."

The above was for 999 files but for my tests showing the pre-allocation taking longer I used 5000 files

I inserted file entries (using SET FILE/ENTER=xxxx rather than copying files) in foward sequence, so file 001_* then 002_* and so on.

Both should have had the same progressive shifting of the EOF marker and using pre-allocation should have removed the need to increase the file allocation 6 times.

[John Gillings]

John,
Fixed the random number generator:

create/dir/prot=(o:rwed) [.test]
TEST.DIR;1 1/208
@time @crefiles [.test] 5000 *
Dirio= 36782 Bufio= 79990 Kernel=1330 RMS= 272 DCL= 57 User= 32 Elapsed= 3912
TEST.DIR;1 410/416
@time del [.test]*.*;
Dirio= 22918 Bufio= 5010 Kernel= 117 RMS= 16 DCL= 0 User= 2 Elapsed= 1736
TEST.DIR;1 1/416

create/dir/prot=(o:rwed)/alloc=1000 [.test]
TEST.DIR;1 1/1040
@time @crefiles [.test] 5000 *
Dirio= 37267 Bufio= 79990 Kernel=1306 RMS= 291 DCL= 73 User= 31 Elapsed= 3886
TEST.DIR;1 425/1040
@time del [.test]*.*;
Dirio= 23419 Bufio= 5010 Kernel= 104 RMS= 17 DCL= 0 User= 1 Elapsed= 1673

These results suggest that inserting in sequence is fastest, reverse sequence is slowest, and random is somewhere in the middle. For these numbers preallocation is not significant (but then it's a maximum of 5 allocations).

BTW, the reason why you might expect preallocation to matter is that directories aren't extended. They're always contiguous, so expanding a directorory involves finding a new continuous chunk of disk and copying the entire contents of the directory to the new extent. As directories get bigger, it costs more and more, especially if you're extending in little chunks.

On the other hand, comparing your stats with mine, maybe you could do with a quicker box? ;-)

[John McL]

John, those are interesting figures for random file generation but what happens with purely sequential entries (i.e. new entries always inserted at the end of the file)?

I'm running on one of a clustered pair of COMPAQ AlphaServer DS20E 666 MHz machines, with a 2-member disk shadow set where one is directly attached to my machine and the other MSCP served from the other machine. (It's just a test machine for low level stuff or infrasturucture code.) That said, I can't see why preallocating the directory size on these disks would be slower than not preallocating it.

[John Gillings]

John,

My "in order" runs are "@time @crefiles [.test] 5000" - roughly 35 seconds.

Inverse order runs are "@time @crefiles [.test] 0 -1 5000", around 41 seconds.

Random order are "@time @crefiles [.test] 5000 *", around 39 seconds.

As I said before, to solve your question, try single stepping your insertions and look at the distribution of entries between each insertion. Try:

$ PIPE DUMP/DIRECTORY TEST.DIR | SEARCH SYS$PIPE "Virtual block number","Name:"

This will generate a map of the distribution of entries in the directory. With a bit of post processing you could make it more visually obvious.

If you single step your entries in parallel, one to a default directory and one to a preallocated one, you can compare the maps to see when they diverge. You can also use the size of the file to determine when the smaller directory gets extended. Going even further, use:

$ PIPE DUMP/HEAD/BLOCK=COUNT:0 TEST.DIR | SEARCH SYS$PIPE "LBN:"

to see where the directory is allocated on the disk.

Here's a command to compare two directory "maps"

$ PIPE (d1=F$TRNLNM("SYS$OUTPUT") ; DEFINE/JOB/NOLOG D1 &D1 ; -
PIPE DUMP/DIRECTORY TEST1.DIR | SEARCH SYS$PIPE "Virtual block number","Name:") | -
(d2=F$TRNLNM("SYS$OUTPUT") ; DEFINE/JOB/NOLOG D2 &D2 ; -
PIPE DUMP/DIRECTORY TEST2.DIR | SEARCH SYS$PIPE "Virtual block number","Name:") | -
DIFF d1 d2

Check $SEVERITY 1 means no difference, 3 means difference.

Do at this between entries and stop when they're different.

[John Gillings]

John,

Oh well :-( there's a flaw in my nested PIPE hack. It doesn't always work for directories with more than a few hundred entries. Some kind of timing issue/deadlock between pipe segments when the output gets too big. Since I'm skipping over the middle segment to DIFF the two outputs, I suppose it's outside the design rules of PIPE.

Recoding to use temporary files, I found the directories remained identical, even after extending the smaller directory. Looks to be the same for in order, reverse order and random. I guess that just means that directory allocation of entries is deterministic, and extends are transparent. Stands to reason, but now verified by experiment.

So you've got yourself a mystery! If the entries are inserted identically, the preallocated directory should be exactly the same work, minus the work of the extends, and should therefore be faster (at least marginally).

[P Muralidhar Kini]

Hi John,

>> It's been suggested (see early in this thread) that preallocating the
>> directory file size will be faster because it won't need to be expanded,
>> which makes sense. So why is it that when I test it - and I have done
>> several times with very similar results - that what in theory should be
>> faster is actually slower on a machine where I am the only user?

By preallocating the directory file size, the operation will be faster only
in the following scenario

- If the directory does not have free space after its current EOF
- disk is badly fragmented and it will take a while to get a new contigious
space on the disk
- and ofcourse, the time taken to move the directory from one portion of the
disk to another

If we create a directory with some random number of files such that above
conditions are not met then tests with and without preallocating the directory
file will give similar results.

A good test case to check the benefits of preallocation would be to have a
setup where all the conditions mentioned above are met. Then the test results
with and without preallocating the directory would give different results.

In the system where you have seen performance problem with the directory
files if the above conditions exist then pre-allocating a directory file is a good thing to have.

Regards,
Murali

[John McL]

Murali,

If there's free space beyond the current allocation and the directory file expands into there then the overhead will be minimal

But, surely pre-allocation shouldn't mean that the same activity - adding files in perfect forward order 0001*, 0002*, 0003* ... etc - takes more time than multiple new allocations?

The attached doc is a Word format version of my summary. If there's no resolution to this mystery in the next 24 hours I'll have to close the thread (after posting a text version of the document) because I'm going on leave for 2 weeks.

[P Muralidhar Kini]

Hi John,

>> But, surely pre-allocation shouldn't mean that the same activity - adding
>> files in perfect forward order 0001*, 0002*, 0003* ... etc - takes more
>> time than multiple new allocations?

This looks odd. How many times have you executed the tests?
Once or multiple times. May be there was some operations going on in the
system at the time you were running the tests with pre-allocation due to
which the time taken appears to be more.

You can run the test scripts (with & without preallocation) multiple times
(say 4 to 5 times) and then take the average time taken for with and
without pre-allocation for comparison.

Regards,
Murali

[John Gillings]

John,

If you can produce a single command procedure that can repeat your entire experiment and post it, several of us could run it in different environments to compare results. I'd be looking for something I can just download and run, rather than downloading numerous pieces and repeating multiple steps.

[John McL]

Murali, I tested CREATE/DIR and CREATE/DIR/ALLOCATED=1000 at least 3 on the same machine and the pre-allocated was always slower.

I've attached a command procedure that will run the test, display the timings and then delete the files and the directory. All that you need to do is create the directory before you start it, and you can opt for with /ALLOCATE= or without.

I've tried to run tests on another machine. The execution time for /ALLOC= and without is around 45 seconds but (a) there's other users on this cluster and (b) the disks are in a SAN disk farm and there may be channel contention from other machines.

[John Gillings]

John,

Interesting...

On my system

HP BL870c (1.59GHz/12.0MB), with 8 cores active
OpenVMS V8.3-1H1
SAN Storage of some sort

Several repeats:

$ create/dir [.big]
Dirio= 12007 Bufio= 65003 Kernel=3346 RMS= 455 DCL= 56 User= 80 Elapsed= 5443
Dirio= 12012 Bufio= 65003 Kernel=2592 RMS= 402 DCL= 54 User= 66 Elapsed= 4459
Dirio= 12007 Bufio= 65003 Kernel=3237 RMS= 378 DCL= 46 User= 62 Elapsed= 5106
Dirio= 12028 Bufio= 65003 Kernel=2892 RMS= 413 DCL= 51 User= 62 Elapsed= 4777

$ create/dir [.big]/alloc=1000
Dirio= 11873 Bufio= 65003 Kernel=2714 RMS= 412 DCL= 52 User= 51 Elapsed= 4575
Dirio= 11873 Bufio= 65003 Kernel=3466 RMS= 385 DCL= 59 User= 56 Elapsed= 5279
Dirio= 11873 Bufio= 65003 Kernel=2253 RMS= 426 DCL= 50 User= 39 Elapsed= 4036
Dirio= 11873 Bufio= 65003 Kernel=2026 RMS= 398 DCL= 44 User= 58 Elapsed= 3840
Dirio= 11873 Bufio= 65003 Kernel=1994 RMS= 404 DCL= 50 User= 59 Elapsed= 3812

Note the preallocated directory has consistently fewer DIRIOs. I've also confirmed that the resulting internal layout of the directories are identical.

I'm rather surprised at the variability of the elapsed times (>20%!), but overall it looks like preallocation has the edge. I'd say the "answer" here is that the unpredictability of timing of the storage subsystem contributes far more variability to the elapsed time than the effect of preallocation (which makes sense to me, but the magnitude is more than I would have predicted).

I don't know how many repeats you've performed, but it's conceivable that you just happen to have got samples in opposite extremes for the different cases. This may have to do with the ordering of your tests. Try multiple repeats (>20) for the same case in sequence, then do the same for the other case. Discard the first few samples for each case on the assumption that they've setup the storage subsystem caches for the workload, then do some proper statistical tests on the data.

The numbers above definitely show that preallocation works, and reduces the I/O load, but the observation is that the saving may not be reflected in elapsed times.

[Hein van den Heuvel]

I use the LD driver to trace all IO to enter 50 long files names in ascending order in an empty directory, starting with 0 blocks, or pre-allocated.
The cluster size on the (100 block) LD device was just 1.
Each directory block could hold 6 entries.
The header for the file to be entered was LBN 19.
The header for the test directory was LBN 18.


Earlier I believe I indicated you could use any file-id for an enter. And I'm sure I tested that (on I83-H1 ?). But on this 8.3 test box that was not the case. I needed a real file, and it re-wrote the header for each ENTER. I think that is/was a bug. I suspect the rewrite if _meant_ to update the hardlink count, if there are hardlinks on the device. But I did not have those hardlinks (no set volu/vol=hard lda3:), and the header rewrite was with exactly the same bytes over and over. Silly! IMHO.

Anyway... here is a partial trace, about 30 files in.

What you see is a last write filling up LBN 49, which was like VBN 5 in the directory.
Followed by the wasteful header write for the file being entered.

23:05:39.53 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.53 00.00 19 512 NORMAL WRITEPBLK|DATACHECK

Next is a block zero for tail 3 block in the newly expanded directory space, followed by a 5 block read from the begin of the old space (44), writing to the begin of the new space (50). The directory header (lbn 18) is update for the new map, and finally the new file is entered (lbn 56 write + the silly bn 19 non-update).

23:05:39.54 00.00 56 1536 NORMAL DSE|ERASE
23:05:39.54 00.00 49 512 NORMAL READPBLK
23:05:39.54 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.54 00.00 55 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.54 00.00 44 2560 NORMAL READPBLK
23:05:39.54 00.00 50 2560 NORMAL WRITEPBLK|DATACHECK
23:05:39.54 00.00 18 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.54 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.54 00.00 19 512 NORMAL WRITEPBLK|DATACHECK

The next entry causes a read of all valid directory block, and the enter game continues.

23:05:39.55 00.00 50 3584 NORMAL READPBLK
23:05:39.55 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.55 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.56 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.57 00.00 19 512 NORMAL WRITEPBLK|DATACHECK

There one more oddity coming.
Watch when we add 5 more files.
As block 56 is filled, block 57 is written ( -1 end marker?) and then a final write to block 56 is done. This happens for each block.

23:05:39.57 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.58 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.58 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.58 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.59 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.60 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.60 00.00 57 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.61 00.00 18 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.61 00.00 56 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.61 00.00 57 512 NORMAL WRITEPBLK|DATACHECK
23:05:39.61 00.00 19 512 NORMAL WRITEPBLK|DATACHECK


Now on to pre-allocated. Same spot in the procedure. Just write and write. When full, jump ahead, skip back and then go ahead working on the next block.

Full log is attached.

Clearly there is less work for pre-allocated.
So either John McL had extreme 'bad luck' reproducing, or there is an interaction (XFC?) which we do not yet understand.



23:08:51.94 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.94 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.95 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.95 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.96 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.96 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.97 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.97 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.98 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.98 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:51.99 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.00 00.00 19 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.01 00.00 50 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.01 00.00 18 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.01 00.00 49 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.01 00.00 50 512 NORMAL WRITEPBLK|DATACHECK
23:08:52.01 00.00 19 512 NORMAL WRITEPBLK|DATACHECK


Enjoy,
Hein

[Hein van den Heuvel]

1000 block LD test device.
Hein

[John McL]

Thanks Hien.

No time now to follow through with this because I'm about to go on leave until May 10th. (Planes are flying to Europe again!)

I'll leave this thread open until I get back because I want to think about what you've said and I need to award points.

Latest - hopefully final - version of ths summary doc is attached. (The only changes are on the last page.) It's in Word format, apologies if this is a problem but I'm out the door in less than 5 minutes.

[P Muralidhar Kini]

Hi Hein,

>> So either John McL had extreme 'bad luck' reproducing, or there is an
>> interaction (XFC?) which we do not yet understand.

I agree with the first part. I dont know why the test results are not indicating
the fact that pre allocating the directory is faster.
There is definitely something else going on.

However I dont think there is any interaction with XFC involved here.
XFC is the data cache in VMS which caches only file contents.
The directory files are still dealt with by XQP itself.

So when creating a directory with or without pre-allocation, XFC would not get
involved. However when files are created within the directory, XFC would get
involved as usual (i.e. as in case for any file on disk).

So I guess, the unknown element is something other than XFC.

Regards,
Murali

[John McL]

Thanks to all. This has clarified ineffeicient and efficient operations with directories when creating and working with large numbers of files.

I hope I've encapsulated all this in the summary document (see attachments above) that other people might refer to in future.