Practical limit on number of files per directory

[Coleman Weber]

Hi,
We have an application that will be placing MANY files per directory. Are there any limits that I should be thinking of or concerned with when I allocated the associated filesystems??? A rough count of all of the individual files is 1,000,000. I thought a database better suited to this task, but the app wants to have directories and individually named files.
Thanks for any suggestions!!!!!!

Doug

[Pete Randall · Accepted solution]

Doug,

See this thread for a worst case scenario and a lot of good input:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe87a42308663d611abdb0090277a778c,00.html


Good luck,
Pete

[James R. Ferguson]

Hi Doug:

See this recent discussion:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xe87a42308663d611abdb0090277a778c,00.html

Regards!

...JRF...

[Pete Randall]

Doug,

Seeing as how you're new and all, you might want to take a look at this link, too:
http://66.216.15.50/ITRCForumsEtiquette/

Enjoy the forums,
Pete

[Wodisch]

Hi Doug,

don't try this (1mio files in one dir), not even at home!

As UN*X writes directories *always* as a whole, not in parts, the bigger the directory is, the slower your system will become.

Try this for demonstration:

terminal 1:
sar -a 1 9999

terminal 2:
mkdir /tmp/toobig
cd /tmp/toobig
n=1
name="very-long-and-I-mean-it-that-way-filename-just-to-fill-directory-even-faster-you-see-what-I-mean"
while [ $n -lt 999999 ];do
touch $name.$n
n=$(expr $n + 1)
done

As the directory grows your system will spend more and more time to read and write the dircetory's contents (always completly), it will become sluggish! Look for the column "dirbk/s" in the sar-output :-(

Got my point?

Wodisch

[Michael Lampi]

The other discussions referenced all have excellent pertinent information.

Tests I have run under HP-UX indicate that the practical upper limit of files in a given HP-UX or JFS directory is well under 100,000.

In fact, even when using a NAS server, such as the FasTraak ones from Traakan, the standard HP-UX tools such as "ls", "find", etc., have problems in directories with sizes of around a million (1,000,000) entries. Some problems crop up even earlier, as the applications and tools are not configured to support such a large number of directory entries.

[Coleman Weber]

THANK YOU to all that gave me feedback and for your time and assistance!!!!!!