topic Re: How to delete large number of files in Operating System - Linux

How to delete large number of files

Raynald Boucher — Wed, 30 Nov 2005 12:29:13 GMT

Good day all,

We have a directory that holds around 1 million files. What is the fastest way to delete all those files short of rebuilding the filesystem the directory is housed in.

I used
"find . -type f -print | xargs /bin/rm"
and it took 20 hours (on a D380).

There must be a faster way...

Thanks

Rayb

Re: How to delete large number of files

Jeff Schussele — Wed, 30 Nov 2005 12:34:09 GMT

Hi Rayb,

Well....the quickest way IS going to be destroying/rebuilding the LV.
The problem is the dir size. With 1 million files *every* lookup and inode mod is going to take a long time - especially on a K-class.
Do you *really* need that many files in a single dir?
If you do then I'd suggest that the dir be an FS & LV unto itself. Then you can knock it down & rebuild it w/o touching anything else.

My 2 cents,
Jeff

Re: How to delete large number of files

S.Rider — Wed, 30 Nov 2005 12:34:51 GMT

If I wanted to delete all files in a directory, I think I would
rm -R
mkdir
adjust owner/group/permissions as required.

Not sure if this would be faster.

Re: How to delete large number of files

James R. Ferguson — Wed, 30 Nov 2005 12:38:07 GMT

Hi Rayb:

Well, as you said, the fastest way is to rebuild the filesystem.

Consider: If you are deleting the vast majority of files, it may be faster to backup what you want to keep; rebuild the filesystem; and reload what you kept.

If you can specify whole subdirectories in the filesystem, I'd certainly use:

# rm -rf /mountpoint/dir1 /mountpoint/dir2 ...

If you need some subdirectories retained, but not others, use a combination of the recursive 'rm' above and a 'find -type f' *limited* to those subdirectories where you want to remove only the files.

Regards!

...JRF...

Re: How to delete large number of files

A. Clay Stephenson — Wed, 30 Nov 2005 12:52:43 GMT

You've already answered your own question. Rebuild the filesystem. Now go find a hammer and hit yourself over the head. 10^6 files in a directory is (stupid,dumb) less than optimal.

We might be able to rearrange the question so that the time required doesn't matter very much:

1) rename the existing directory
2) create a new directory with the name of the old
3) start your rm process on the renamed directory in the background - nohup'ed.

The idea is that you have a fresh new directory available for use while the old one is being cleaned up and the cleanup time is not nearly so critical.

Still, Plan A should be divide and conquer so that you never have that many files in a directory. In general, filesystems are poor substitutes for databases.

Re: How to delete large number of files

Steven E. Protter — Wed, 30 Nov 2005 13:23:49 GMT

Shalom Raynald,

To sum it up, you need a plan that does not leave you 1 million or even 100,000 files in the same directory.

Thats what subdirectories are for.

SEP

Re: How to delete large number of files

Raynald Boucher — Wed, 07 Dec 2005 09:53:11 GMT

Thanks for your answers.
I fully agree that it's too big but it's a legacy and I, personally, can't do much about it at this moment.
One good thing, this directory is the only thing on that file system.

Does anyone know of a faster way to back it up/ restore it than using "ftio"?

Thanks

Re: How to delete large number of files

A. Clay Stephenson — Wed, 07 Dec 2005 10:01:20 GMT

Unless you look at a commercial backup product like Data Protector, fbackup is probably your best bet. Configure it with multiple reader processe.

Re: How to delete large number of files

Raynald Boucher — Wed, 07 Dec 2005 10:12:59 GMT

We tried.
It fails on an excessive file list.
That's why we use ftio.

Thanks

Re: How to delete large number of files

Raynald Boucher — Wed, 07 Dec 2005 10:13:50 GMT

Thanks all