topic Re: Maximum Directory Entries in Operating System - OpenVMS

Maximum Directory Entries

Robert Atkinson — Fri, 02 Jul 2004 05:36:12 GMT

I know there's a maximum number of files you can put on a disk (SHOW DEV x /FULL), but does anyone know if there is a limit on the maximum number of entries in a directory, and how this is calculated?

Cheers, Rob.

Re: Maximum Directory Entries

Mobeen_1 — Fri, 02 Jul 2004 05:40:19 GMT

Rob,
I don't think there is any such limit...in directory...i have never come across this in my stint in VMS

The only limit on files is on the disk and that is " Maximum files allowed" when you do a sh dev/full

regards
Mobeen

Re: Maximum Directory Entries

labadie_1 — Fri, 02 Jul 2004 05:40:42 GMT

the max number of files is related to the /maximum_files and the /header qualifiers of the $ initialize command

I think that a directory, IF Vms has enough room to increase the size of a directory file while being contiguous, does not have a short limit.

See help/message diralloc

You have had this error message recently ?

Re: Maximum Directory Entries

Ian Miller. — Fri, 02 Jul 2004 05:44:43 GMT

max number of files in a directory is usually limited by finding enough contiguous space for the directory file. Large directories are inefficent to access.

Is there a problem you are trying to solve?

Re: Maximum Directory Entries

Robert Atkinson — Fri, 02 Jul 2004 05:46:38 GMT

Nope.

But I do have an appliction, WebReports, that writes all the PDF's to a single directory.

We've only just started rolling it out and there's already 10,000 files in there!

Obviously, for performance reasons, I'm going to have to start breaking this up, but I didn't want to be caught out before getting round to it.

Rob.

Re: Maximum Directory Entries

Wim Van den Wyngaert — Fri, 02 Jul 2004 05:47:18 GMT

Good question.

http://groups.google.com/groups?hl=nl&lr=&ie=UTF-8&threadm=Pine.LNX.3.96.980511150427.31835N-100000%40irys.stanpol.com.pl&rnum=3&prev=/groups%3Fnum%3D100%26hl%3Dnl%26lr%3D%26ie%3DUTF-8%26as_drrb%3Db%26q%3D%2Bin-a-directory%2Bvms%2B%2522maximum%2Bnumber%2Bof%2B%2522%26btnG%3DZoeken%26as_miny%3D1997%26as_minm%3D5%26as_mind%3D12%26as_maxy%3D2004%26as_maxm%3D7%26as_maxd%3D2

Re: Maximum Directory Entries

Mobeen_1 — Fri, 02 Jul 2004 05:47:57 GMT

Rob,
I am sure you would have heard of these too..

Version Limits on files

/MAXIMUM_FILES in init

Since i don't see any option for this while creating a directory, i am assuming that it may not be present (CREATE?DIR)

I would love to see what other people think on these forums.

regards
Mobeen

Re: Maximum Directory Entries

Mobeen_1 — Fri, 02 Jul 2004 05:49:46 GMT

Wim,
Thanks for that link :-)

Looks like so far the answers to this question have been in & around whats been discussed in the link posted by you

regards
Mobeen

Re: Maximum Directory Entries

Ian Miller. — Fri, 02 Jul 2004 05:50:30 GMT

Sounds uncomfortable :-)
You may find DFU (V3.0) DIRECTORY commands useful

Re: Maximum Directory Entries

Wim Van den Wyngaert — Fri, 02 Jul 2004 06:00:22 GMT

I started a test.

But I noticed that create fails when someone does a EVE of the directory. Old sickness of VMS ... other threat ...

I'll keep you informed

Wim

Re: Maximum Directory Entries

Ian Miller. — Fri, 02 Jul 2004 06:35:53 GMT

"But I noticed that create fails when someone does a EVE of the directory. Old sickness of VMS ... other threat ..."
probable read lock on the directory file preventing update of the directory to add new entry for created file.

Re: Maximum Directory Entries

Hein van den Heuvel — Fri, 02 Jul 2004 09:01:30 GMT

The is no hard limit.
There TENDS to be a performance degradation.
In older VMS versions there was a performance knee at a directory size of 128 blocks when doing wildcard lookups.
If filenames are generated 'in oder' with ever increasing names, there is very litle overhead indeed.
If the names are random, then new files will frequently cause the needed to 'shuffle' up a good chunk of the directory to make room for the new name.
Keep names short if you can!
bad: [report_directory]adobe_report_for_july_05_2004_00546.pdf
good: [report_directory]2004070500546.pdf

Divide and conquer:
better:[200407_reports]0500546.pdf

hth,
Hein.

Re: Maximum Directory Entries

Robert Atkinson — Fri, 02 Jul 2004 09:03:49 GMT

That's odd...got this from Google earlier, which implies Random names are better :-

With normal random file naming behavior, directory shuffles are infrequent. However, non-random behavior can cause problems. A classic case is a DELETE *.*;* on a big directory. The file wildcarding of course returns the files front to back, and so files are deleted from the front of the directory - precisely the worst order. So the time to delete all the files in a big directory goes with the square of the number of files. If the directory is really huge, it's well worth building a command procedure to delete the files back to front.

Re: Maximum Directory Entries

Willem Grooters — Fri, 02 Jul 2004 09:20:29 GMT

Rob,

Ian is right:

max number of files in a directory is usually limited by finding enough contiguous space for the directory file.

You can run into "funny" behaviour if there is too litlle contiguous space to hold an expanded directory file. Especially if free space is scattered. Browse down the forum for some example....

To keep your file collections usable, I gues you wouldn't want 10.000th of files into one directory - let alone it _may_ introduce a performance penalty.

Willem

Re: Maximum Directory Entries

Kris Clippeleyr — Fri, 02 Jul 2004 09:26:43 GMT

Robert,

So the time to delete all the files in a big directory goes with the square of the number of files. If the directory is really huge, it's well worth building a command procedure to delete the files back to front.

For deleting "large" directories, I find that DFU comes in handy.
Oth, to prevent any application that writes its files in one directory from "over-filling" a certain directory, you can set up a search list of logical names, that point to different physical directories, and at certain time intervals rotate the definition. Writing will happen to the first directory in the list, reading will be tried on all.
E.g.:

$ DEFINE LOG DISK2:[LOG1],DISK2:[LOG2],DISK2:[LOG3]
$ CREATE LOG:T.T
ctrl/Z
$ DIREC LOG

will show T.T in DISK2:[LOG1]

hereafter

$ DEFINE LOG DISK2:[LOG2],DISK2:[LOG3],DISK2:[LOG1]
$ CREATE LOG:X.X
ctrl/Z
$ DIREC LOG

will show T.T in DISK2:[LOG1], and X.X in DISK2:[LOG2]

We used this trick once on an application that created a huge amount of uniquely named logfiles.

Greetz,

Kris

Re: Maximum Directory Entries

Wim Van den Wyngaert — Fri, 02 Jul 2004 10:36:29 GMT

First test results on a 2 node 4100 cluster using 7.3.

DCL loop to create files with name starting with 0-9 + fixed name. Started 2 jobs on each node to fill the directory.

After almost 5 hours :
First hour : 4320 files created
Second hour : 3490 files created
Third hour : 2732 files created
So : big directories are slow for file creation.

De directory file is now 7000 blocks and contains 22.000 files.

Test continues wednesday.

Re: Maximum Directory Entries

Hein van den Heuvel — Fri, 02 Jul 2004 16:53:03 GMT

In reply to my reply Robert wrote:

" That's odd...got this from Google earlier, which implies Random names are better :-

With normal random file naming behavior, directory shuffles are infrequent."

Define 'normal'!?

I was assuming (may well be wrong) that the applcation in question just kept on adding files. Deleting was not mentioned (yet). If you just keep on adding, then adding at the end is best. This will cause the directory to grow at least by a cluster and maybe more.

The 'normal' behaviour referred to is probably to have files coming and going over time maybe a few more coming than going. Then linear naming is horrible, when doing the deletes. The adds will be fine, but the deletes will always be from the beginning (first directory block) and every directory block emptied will cause a shuffle down.

A random delete will unlikely empty a block, so will not cause a shuffle. Hopefully it will create enough room for a future random add for a different file, but targetted to the same block

btw... for totally optimal directory packing I forgot to mention dropping 'obvious' file types. Like '.pdf' in a report directory. Just give the exception files an extention. And that extention can be ".F" for fortran source and ".O" for objects... if performance is more important than clarity and easy of use (unlikely!)

Ramblings...

For computer named/used files, where humanoids are not reading any meaning into the file name (more or less like VMSmail extention files), the optimal packing is ofcourse by using a base-36 (or worse!) characterset: 0-9A-Z. Using that, 2 characters can identify 1000+ files, 3 is good for almost 50,000 and 4 can address more than a million, but youhave to add 14 bytes of directory data per entry.
10,000 files woudl then just take 350 blocks of .DIR file. Crazy but possible.

Hein.

Re: Maximum Directory Entries

Andy Bustamante — Tue, 06 Jul 2004 10:50:59 GMT

As someone already mentioned the 128 performance block hit for directory size was lifted in VMS 7.2.

You haven't mentioned delete or long term storage. Are these these files out of date in after being displayed or do you use long term storage? For short term use, I use a six directory logical for our web server's reports and a batch job that redefines it in 10 minute intervals. Reports are available for at least 50 minutes, with the sixth directory having the contents deleted once an hour. For very busy sites, distribute the directories among multiple disks.

Re: Maximum Directory Entries

Robert Atkinson — Wed, 07 Jul 2004 01:59:11 GMT

The reports will be kept for at least 6 months, longer if I can find the disk space.

Rob.

Re: Maximum Directory Entries

Wim Van den Wyngaert — Wed, 07 Jul 2004 09:50:15 GMT

Test results continued but only 2 jobs instead of 4.

There are now about 35.000 files in my directory that itself is now 11.000 blocks.

No problems yet but :

1) doing dir/siz test.dir takes between 2 and 10 seconds (why ?)

2) doing dir/tot/sin=15:30 takes about 10 minutes (acceptable, old machine)

Wim