Max number of files per filesystem or directory?

[Mareike Ernst]

Hello all,

One of our developers popped a question I couldn't quite answer:
Is there any limit on the number of files a directory or filesytem can contain?
(No, I'm not talking inods, I know how to check that).
Does vxfs impose any limits?
We're also running Serviceguard, but I don't suppose that should make any difference.

Thanks,
Mareike

[Robert-Jan Goossens]

Hi Mareike,

The limit is the filesystem size. There is no inode limit on a JFS filesystem.

Kind regards,
Robert-Jan

[Robert-Jan Goossens]

add on,

JFS has an unlimited number of inodes per file system. They are dynamically assigned.

Document description: What are the differences between HFS and JFS file systems?

http://www4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062909602

[Pete Randall]

Mareike,

While there is no real limit on the number of files, there is a practical performance limitation. When you get up to millions of files in a relatively flat hierarchy, performance can degrade significantly. Something to keep in mind.


Pete

[Dave Olker · Accepted solution]

Hello Mereike,

While VxFS does not impose a limit, there are some recommendations for acheiving optimal VxFS performance, especially in the area of the maximum number of files to store in a single *directory*.

To quote from Mark Ray's "JFS Tuning and Performance" technical paper:

_____________________________________

Large Directories

One myth is that a file system with a large number of small files (for example 500,000 files which are 4kb in size or less) does not need to be defragmented. However, a file system with thousands of small files is likely to have very large directories. Directories are good examples of files that are often opened, extended, and closed. Therefore directories are usually fragmented. â Smallâ is a relative term. The number of blocks taken by a directory also depends on the size of each file name. As a general rule, consider a small directory to be one that has fewer than 10,000 directory entries.

When adding a new file to a directory or looking for a non-existent file, every directory block must be search. If the directory has 1000 directory blocks, then the system must do at least 1000 I/Os to add a single file to the directory or search for a non-existent file. Simultaneous directory searches also incur contention on the inode, extent map, or directory blocks, potentially single-threading access to the directory. Long delays can be detected when
doing multiple â llâ commands on a single large directory.

With JFS 3.3, if the file system is upgraded to the version 4 disk layout, then large directories can be defragmented when doing an extent reorganization, so that the directory contains larger but fewer extents. The reorganization of a directory can relieve bottlenecks on the indirect blocks when large directories are searched simultaneously.
_____________________________________


I highly recommend reading the entire technical paper, which is available for download from:

http://docs.hp.com/hpux/onlinedocs/5576/JFS_Tuning.pdf

Re