file system size and system performance relationship

[Jeongbae Min]

Hello,

our system admin define a huge 12 T byte file system using EMC NS40. I think it cause system bottleneck to backup data using backup library as well as maintain big file system.

Does anyone have good white paper regarding file system and system performance? or someone can give me some advice what is the proper size of filesystem?

Most of users use 1.2 Gbyte flat files to design chips.

Thank you in advance,

Jeongbae

[James R. Ferguson]

Hi:

Large filesystems do not have performance problems per se. A large number of files in a filesystem (regardless of size) where searches ('find's) span the whole extent of the filesystem's root directory _are_ costly to performance.

The mount point options you use and the way you use a filesystem (e.g. many additions and deletions; frequent metatdata changes) can help or hinder performance.

One nice white paper on JFS filesystem performance and tuning can be found here:

http://www.docs.hp.com/en/5576/JFS_Tuning.pdf

Regards!

...JRF...

[Bill Hassell]

The total space for the filesystem is not important so there is no 'proper' size. What is important is the quantity of files and directories at a single level. You can create twenty 500GB files and the filesystem will be almost full. But accessing the filesystem with tools like ls or du will very fast, just like smaller filesystems. But if there will be thousands or millions of files in one directory, interactive tools like ls or find will appear to be very, very slow. They are processing file names just as fast as a small directory, but there are 100 times or even 1000 times more content, thus much slower to report results. And of course, using "*" as a filename filter, especially as input to a backup program like tar will fail with: line too long.

However, 1.2 GB files will be a good fit, especially when segregated into lots of directories such as projects or user IDs. Most commercial backup programs handle small and large files quite easily. In your example, the GB files will be easy to backup.