Maximum Directory Entries

[Wim Van den Wyngaert]

The 20 seconds is when using a new block of the dir file.
When the 108 blocks must be allocated, the direcory is blocked for 30 seconds.

[Wim Van den Wyngaert]

Ending the test. The penalty is now over 30 seconds and still increasing (117.000 files).

It is clear that big directory files can not be used by applications that frequently create files or by real time applications.

Now let's see how long delete will need ...

Wim

[Wim Van den Wyngaert]

Delete was going very slow (slower than creation).

I tried DFU 2.6 delete dev:dir/dir but I get

%DFU-I-PARSEDIR, Parsing directory tree OPS$MGR:[WVW.TEST]*.DIR;1
%DFU-W-NOSUBDIR, no subdirectories found in this tree

%DFU-I-CLEANUP, Deleting OPS$MGR:[WVW]TEST.DIR;1...
%DFU-E-READERR, Error reading directory OPS$MGR:[WVW]TEST.DIR;1,
%SYSTEM-F-BADPARAM, bad parameter value
%DFU-E-NOTDEL, Error deleting file TEST.DIR;1 ,
%SYSTEM-F-DIRNOTEMPTY, directory file is not empty

%DFU-I-READY, DELETE command ready

[labadie_1]

for deleting huge directories, there was on the french decus site a tool destroy which was very fast.
I will see if I can find it.

[Wim Van den Wyngaert]

Tried dfu 2.7a and 3.0 (linked myself on 7.3).
All have the same problem.
The cluster will know what to do this WE.

Wim

[Hein van den Heuvel]

Hmm, sounds like you found a DFU problem. I'll send Ton Dorland a mail alerting him to this thread.

Instead of deleting, can you save whatever is valuabel on the disk, and re-init the disk? Much faster!

If you do dedice on the delete *.*.* then be sure to monitor its progress (I know you will), because it may 'never finish', that is it might take more than a weekend.

You may want to try my script below. It is from the days before I switched to perl :-). It just feeds delete a number of files in reverser order over and over again. You may want to tweak teh command line length (there is a test for 200 in there).
The reverse order will prevent the shuffling when a directory block empties. The multiple files somewhat reduce the delete image activiations.

fwiw,
Hein.


$if p1.eqs."" then goto help
$directory/out=sys$Login:delete.tmp/col=1 'p1
$sort /key=(pos:1,siz:-1,desc) sys$Login:delete.tmp sys$Login:delete.tmp
$open/read/error=clean_up sorted sys$Login:delete.tmp
$init:
$read/end=clean_up sorted f
$l2 = f$len(f)
$if f$loc(";",f).eq.l2 then goto init
$files = f
$loop:
$read/end=done sorted f
$l1 = f$len(f)
$if f$loc(";",f).eq.l1 then goto loop
$if l2.lt.200
$ then
$ files = files + "," + f
$ l2 = l2 + l1 + 1
$ else
$ delete 'p2' 'files'
$ files = f
$ l2 = l1
$endif
$goto loop
$done:
$delete 'p2' 'files'
$clean_up:
$close/nolog sorted
$delete sys$Login:delete.tmp;,;
$exit
$help:
$type sys$input

Usage:

P1 (required) = wild card file spec, with optional DIR selection qualifiers
P2 (optional) = delete command qualifiers (/LOG or /CONFIRM)

$exit

[labadie_1]

it seems to me that

$ bac/lo/del disk:*.*;* nl:

is fast, but I have not tested.

I think Hein method should be faster.

[Wim Van den Wyngaert]

Hein,

I was just going to write the same script.
But then I remembered something undocumented : set file/nodir.

So :
$ set file/nodir test.dir
$ del test.dir;
$ def sys$output nl:
$ def sys$error nl:
$ anal/dis/rep dsa1:

It is now running since 5 minutes. Curious how long it will take ...

Wim

[labadie_1]

sorry, typo

bac/lo/del dir nl:a.bck/sav

[Hein van den Heuvel]

uh... what do you think anal/disk is going to do? It is going to consider all those files lost and re-enter them in the lost and found directory. You will just have moved the problem to a new directory
:^).

(btw... I have a 'set file /dir' tool (set_dir) on some freeware (V4?) in the rms_tools directory if you like to undo the set file/nodir :-)

Groets,
Hein.

Stuur me eens een email: hein apestaartje hp punt com.

ps... heb jij een ligfiets?

[Wim Van den Wyngaert]

You're right Hein. I tried it and found that after about 100 messages the syslost directory was still empty. Wrong conclusion.

But if you stop de anal/dis/rep after 15 minutes, you can delete them a lot faster in syslost.

En ja, ik heb 3 fietsen.
1 Gazelle voor met de vriendin te gaan rijden
1 Twister LX voor op te liggen langs de waterwegen (Schelde, Ruppel, Nete en kanalen)
1 Bulls 7.8c voor in de Ardennen af te gaan zien (op asfalt)

Wim

[Wim Van den Wyngaert]

Labadie, yes your delete is a lot faster.

Wim

[Mark Hopkins_5]

A couple of notes about dir performance and
then a small hack for fast deletes.

It doesn't surprise me to see 15 second or
longer delays extending a large directory.
Not only do you have to find the 30k blocks
of contiguous space (which may mean trolling
the entire bitmap), but you then have to
copy the entire directory to its new
location - all 30k blocks in the example
here. The original space needs to be
deallocated and the header updated.

Note that the create/dir command allows you
to allocate the space up front so you don't
have to endure the frequent extends for
directories expected to be very large.


While the old RMS 128 block directory
performance hit was removed, there is still
an XQP knee. The XQP keeps a small index
into the directory using the first few
characters of the filename in each block. For
small directories, a file lookup gets
directly to the correct block in the
directory file and begins a search there.
The index is limited in size (one block)
and as the directory gets larger, the
number of characters used for the index
gets smaller. At a few thousand blocks,
we're down to the first 3 characters of
the file name. If your filenames are
not different in the first 3 characters,
then the entire directory has to be
scanned (there may be a binary search here,
I haven't looked at this code in a long,
long time). The mail directory is a good
example of what not to do - file names of
the form -
mail$nnnnnnnnnnnnn.mai.

The directory shuffle operation used to be
very, very painful. It was done one block
at a time. It is currently done 127 blocks
at a time (last time I looked) and so is
much faster. It is still an n^2 operation
and I can imagine you still see delays on
the shuffle on very large directories.


Aside from DFU or a special program, the
fastest way I've found to delete all files
in a directory is to use anal/disk - you
were so close. Create a file consisting
of a line of 'd' for each file in the
directory -

$ create ans.x
d
d
d
d
^z
$ set file/nodir foo.dir
$ define/user sys$command ans.x
$ anal/disk/repair/confirm disk$xxx
- lots of messages about lost headers -
$

Note that ans.x must have more 'd's than
the number of files you're about to toss.
Also note that lost files you might like
will also disappear as well. As I recall,
it also helps to define sys$output to nl:

This is generally much, much faster than
even deleting from the rear of the directory
even from a dedicated program.

Mark

[Wim Van den Wyngaert]

Mark,

How do you explain that a big wait is done every time 1 block of the directory is taken into use ? (e.g. 20 seconds foir extending the directory, 15 seconds each time 1 block of it is used)

[Ton Dorland]

I will have a look at the DFU BADPARAM error message. I never did test DFU which such a large directory, so I'll try to set it up.

Regards Ton (DFU Author).

PS There will be a new DFU (V3.1) distributed on the next freeware.

[Ian Miller.]

Ton - any chance of DFU being in VMS by default? I'm a big fan of DFU and install it on every system and it would be easier if it was already there.

[Wim Van den Wyngaert]

Ton : hope you make it for VMS 7.*, so I can use it on my systems too.

In any case, Marks solution for deleting the files worked. It took 2 cpu minutes to delete all files (didn't check wall time).

Hope that Mark gets his points Rob. And sorry for all the post you got. But I think the test was useful.

Wim

[Ton Dorland]

Good and bad news. I found the bug, this happens (only) on directories > 32768 blocks.
The good news is that I'll fix it with DFU V3.1. But given the low frequency of this issue, I will not make a fix for previous DFU versions, and thus it won't be fixed for previous VMS versions (thats the bad news).
DFU V3.1 requires OpenVMS Alpha V7.3-1 and higher, or Itanium 8.1 and higher!

Regards, Ton

[Jay Newman]

Mark,
Sorry to butt in but I have been trying to dig my production environment out of a similar (but smaller) mess, where roughly 18,000 files were being deleted every day, and then re-created in the same directory using a date stamp at the beginning of the file name.
I.E., each day's files would be for today's date, "20040713_" followed by the more unique information (location, function, etc.)
I broke this up into 8 directories to get the directory size down somewhat, but I still need to drill down further, especially given your description of how the XQP works with an index.
Can you possibly provide an algorithm relating the size of the directory to the number of characters per file name in the index?
Regards,
Jay

[Hein van den Heuvel]

Seems we are drifting away from the original topic, and overloading this discussion topic.

Jay, why don't you re-enter your question as a fresh topic, with a link back to this one?

Just a suggestion,

Hein.

[Wim Van den Wyngaert]

Ton,

I relinked your dfu on 7.3 and this seems to work. Is it allowed to relink and if so, down to what version ?

Wim

[Ton Dorland]

DFU V3.0 has support for hardlinks. This won't work on versions before VMS 7.3. You will be able to relink on 7.2-X, but it will not run properly.

So you need V2.7 on previous VMS versions.

Regards, Ton

[Richard W Hunt]

First, consider sysgen parameter ACP_DIRCACHE, which governs the size of caches used for directory operations. If this is too small, you are whacked hard because you thrash the crud out of your disk directories.

Second, a place where I see this problem is in the ABS directories. ABS has to be one of the worst-written records managers I have ever seen, or else the documentation is the worst-written documentation, 'cause our Ops guys have spent whole careers trying to keep the ABS directory clean and they can't do it.

I remember having to do some tricksy deletes in an ABS directory when we were prepararing to migrate from VAXen to Alphas. At least some of the tapes were still good, so we couldn't just toss everything into the round file. Though I was sorely tempted. The batch job that trimmed the directory took literally days because we couldn't INIT the disk, we had to preserve it.

[Martin P.J. Zinser]

Hello Rob,

search list logicals are a great way to present multiple physical directories with reports as one resource to applications reading the reports. We use this in our own environment.

Still you either need a "cooperative" writer that creates the reoports in various directories or you need to write a procedure that picks the reports up in the initial direcotry and distributes them from there. What to do pretty much depends on your control over the application.

Gretings, Martin

[Robert Atkinson]

Martin - as usual an excellent solution!

Very little change to the application, but will help break up the directories.

Cheers, Rob.