7 million files and backup

[Sundar_7]

Hola folks,

I have a filesystem which is not too big (100GB) but has approx 7 million small small files.

This obviously has a performance impact in the filesystem also on the backups.

I am sure this is not very uncommon. Will increasing vx_ncsize (and thus inturn increasing ncsize) help the performance ? - any other suggestions ?

TIA

- Sundar.

[harry d brown jr]

RAW filesystem backups!

makes restores a pain, but at least the data is backed up.

live free or die
harry d brown jr

[Sundar_7]

yeah that is what we are doing now.

We would be content with the RAW backup, if it was just a pain, but it was plain useless.

We did a test restore from the RAW backup and a fsck marked 24 K files as invalid. Since this filesystem is very dynamic with the superblock changing every minute, RAW backup is not an option.

[A. Clay Stephenson · Accepted solution]

You will see some minimal improvement from increasing vx_ncsize as well as vxfs_max_ra_kbytes but you really already know the answer to your problem. You need to break this into multiple filesystems. While you will still be reading the same total number of files, you can now break this into multiple reader processes handling different portions of the same tree. You will also benefit from using a "real" backup solution such as DataProtector which lends itself to divide-and-conquer approaches.

[Bill Hassell]

As Clay mentioned, you cannot use classic backup tools as they are just too slow. Unless you use DataProtector, the only alternative is to use fbackup with a mandatory config file to control the number of reader processes (you need all 6). And of course, if you have a very high speed tape drive, it may be starved for data which reduces backup speed by 50:1 or more.

[Dave Wherry]

Of those 7 million files, how many are actually being modified. If they are not changing, do they need to be backed up frquently?
I would look at breaking out the oldest, least accessed/modified files into another file system. Do a full backup of that and then incrementals.