Operating System - OpenVMS
1748034 Members
5060 Online
108757 Solutions
New Discussion юеВ

Re: Maximum Directory Entries

 
SOLVED
Go to solution
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

Test results.

60.000 files in directory file of 19.000 blocks.

File creation with names starting with abc takes 0.1 second but every 5 files it takes 10 seconds. And the directory file is locked during that interval.

Jobs doing the dcl create in a loop pagefault at a rate of 75 per second (initial working set of 5800 pages, used about 1700).

Files starting with z have the same results.

Nice feature for a real time application ...


Wim
Willem Grooters
Honored Contributor

Re: Maximum Directory Entries

Wim,

I think your observations can easily be explained:


1) doing dir/siz test.dir takes between 2 and 10 seconds

2) doing dir/tot/sin=15:30 takes about 10 minutes (acceptable, old machine)


I guess this has to do with caching.
The directory-file may need to be reread several times. Don't forget that access to indexf.sys is involved as well - for getting the size.


File creation with names starting with abc takes 0.1 second but every 5 files it takes 10 seconds. And the directory file is locked during that interval.

Jobs doing the dcl create in a loop pagefault at a rate of 75 per second (initial working set of 5800 pages, used about 1700).


Quite possible that each 5th file will need an extension of the directory file - and new (contiguous!) space has to be searched and allocated for the directory! And again, this requirs quite some extra IO.

It makes sense to do some redesign if timing gets critical.

Willem
Willem Grooters
OpenVMS Developer & System Manager
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

Willem,

Now I have 1 job active doing the creates and the directory file is about 20.000 blocks. The file names start with Z and are about 60 characters long.

dir/siz=all takes 15 seconds when the directory is expanded (108 blocks). These 108 blocks don't explain the delay of 15 seconds. There must be some reshuffling that takes time.

Files creation time is 0.5 - 2 seconds (why ?), thus longer than during the previous trial. The 15 seconds now only happen every few hundred files !

The pagefaults have increased to 700 per second !!!

Wim

Wim
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

size 24.000 blocks, 80.000 files
blocking time : 20 seconds when allocating 108 blocks.
Wim
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

size 33.000 block, 113.000 files

Pagefaults down to 200/sec.

Started a create starting with mmm.
20 seconds blockage every 5 files.

Started a create starting with a123.
22 seconds blockage every 5 files. Sometimes a file creation takes 0.1 seconds, sometimes 3 seconds.
Wim
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

The 20 seconds is when using a new block of the dir file.
When the 108 blocks must be allocated, the direcory is blocked for 30 seconds.
Wim
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

Ending the test. The penalty is now over 30 seconds and still increasing (117.000 files).

It is clear that big directory files can not be used by applications that frequently create files or by real time applications.

Now let's see how long delete will need ...

Wim
Wim
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

Delete was going very slow (slower than creation).

I tried DFU 2.6 delete dev:dir/dir but I get

%DFU-I-PARSEDIR, Parsing directory tree OPS$MGR:[WVW.TEST]*.DIR;1
%DFU-W-NOSUBDIR, no subdirectories found in this tree

%DFU-I-CLEANUP, Deleting OPS$MGR:[WVW]TEST.DIR;1...
%DFU-E-READERR, Error reading directory OPS$MGR:[WVW]TEST.DIR;1,
%SYSTEM-F-BADPARAM, bad parameter value
%DFU-E-NOTDEL, Error deleting file TEST.DIR;1 ,
%SYSTEM-F-DIRNOTEMPTY, directory file is not empty

%DFU-I-READY, DELETE command ready
Wim
labadie_1
Honored Contributor

Re: Maximum Directory Entries

for deleting huge directories, there was on the french decus site a tool destroy which was very fast.
I will see if I can find it.
Wim Van den Wyngaert
Honored Contributor

Re: Maximum Directory Entries

Tried dfu 2.7a and 3.0 (linked myself on 7.3).
All have the same problem.
The cluster will know what to do this WE.

Wim
Wim