Maximum No. of files per filesystem for best performance

[Has]

Hi,
I've filesystems with millions of small files, and we are having peformance issue. When I run ll command it takes forever.
Please let me know what is the max number of files per filesystem for best performance and will it help to distribute the files in different directories within the same filesystem.

Thanks
Has

[RAC_1]

The answer is it depends. Depends on what you are running on your system-apps, databses and anything else. Haing different FS will help to certain extent.

[Steven Schweda]

For the _best_ performance, one file is the
maximum. For _acceptable_ performance, a
higher number is normally allowed.

> [...] will it help to distribute the files
> in different directories [...?]

Are all the files in one directory now, or is
your slow "ll" recursive? Depending on what
you're doing, distributing the files into
(many) multiple directories can probably help.

If you have a program which creates this
mess, it should probably be re-designed.

[Mobeen_1]

Has, I would recommend you to look into the following

1. Spread it across multiple directories
2. Better if you could put these files across different file systems and
3. Best solution would be to relook at the application that is creating these files.

I know that some manufacturig applications create embedded directories and files....in millions. If you are using backup like TSM for this, then it will be a miss. If you think that the time taken to backup is very long...then try restoring...the same :)

regards
Mobeen

[A. Clay Stephenson]

Easy question. 1. Now rephrase your question into something more reasonable and the general answer is that it really isn't the maximum number of files per filesystem but rather how many files per directory. It soulds as though you are using the directory tree as a database substitute. Bear in mind that when a directory change is made (adding a files, deleting a file) that the directory has to be locked so that all processes get valid data, if these changes are freequent, it can lead to long delays. Moreover, directory searches are a linear search so the average access time is proportional to n / 2. As a rule of thumb, if you have to ask is a directory too big, it is.

You may truly find that rewriting your application as a database is a far better answer but in any event, you need to divide your files into many smaller directories.

[Bill Hassell · Accepted solution]

Because you have millions of files, ll will indeed take a very long time. It will run much faster if you simply put the files in different directories. It does not have to be a different filesystem, just keep the number of files in a single directory to just a few hundred.

[Dave Olker]

Hi Has,

Just curious... what OS are you running on this system and what filesystem type are you using? VxFS 3.5? 4.1?

Regards,

Dave

[Has]

Thanks for your replyes
I have HP-UX 11.23,
The operation I am doing on the files are: ls ,awk ,perl ,cat ,grep .. etc.
the number of files I have are 3000000 ,on 500 GB filesystem without directories.

What is your recommendation to have acceptable performance.

Thanks

[A. Clay Stephenson]

Small wonder that you are having performance problems. The problems are a direct result of your design. I would say that as an absolute maximum do not exceed 10000 files per directory and preferably limit the number of files per directory to less than 1000.

[Steven Schweda]

> What is your recommendation to have
> acceptable performance.

Step 1. Define "acceptable performance".

Step 2. Try some things and see what happens.

Step 3. If satisfied, stop. Otherwise, go
to step 2.

[Sorrel G. Jakins]

The same answer as before:

1. Redesign your application to put the 3M things into a database.

2. As a short term fix design a directory tree structure and put in a couple of hundred files per directory. You can set a script to list the directory and then measure 200, 300, 400, 500 per directory to see where the response time shows a marked increase, then back off 50 or 100 and that sets the max # files per directory for you.

3. Go back to #1. You will save yourself heartburn in the future.

Sorrel Jakins

[Dave Olker]

Hi Has,

The reason I was asking about OS version and VxFS version is on 11.23 you have the option of upgrading to VxFS 5.0. VxFS 5.0 has some pretty significant performance enhancements for large directories. I've seen these performance benefits first hand in benchmarks.

For example, one benchmark that creates 40,000 files in a directory, stats them and deletes them, running on VxFS 3.5 took over one minute to complete. When the same test was run on the same system in a filesystem using VxFS 5.0 and disk layout 7 the test completed in under 10 seconds.

Bottom line, you can solve your problem in numerous ways, and there will always be recommendations for how many/few files to put in a single directory. But I wanted to let you know that VxFS 5.0 is strongly recommended by HP for use with large directories and you may find that upgrading your VxFS version and disk layout version might give you enough of a performance relief that you don't need to change your application.

Something to consider...

Dave

[Mobeen_1]

Has, i agree with many of the posts prior to this

From your second question the bottleneck is very evident and it would not be a surprise that you are having performance problems :)
-----------------------------------
The operation I am doing on the files are: ls ,awk ,perl ,cat ,grep .. etc.
the number of files I have are 3000000 ,on 500 GB filesystem without directories.
-----------------------------------------

For better performance, i would suggest that you change your operation/app/what ever you are using...and divide your total no. of files into say 10 directories....then you look at the performance...if you are not satisfied....then refine it further...

Basically..what i am suggesting is...create some directories say 10....and then have 300000 files in each of the directories...and look at the performance...if this doesn't satisfy...refine it further.

regards
Mobeen

[Mobeen_1]

To add to my earlier post...try the way Steven has suggested....start with something....on this route until...you get what you feel is acceptable performance

regards
Mobeen

[Has]

Thanks everybody , I appreciate your help

[Has]

Thanks everybody