Finding and deleting file

[Henry Chua]

Hi Guys,

In your opinion, what should be the faster way to locate and delete a file. I am currently using "find" and "rm" command using a shell script. But it is taking very long. Any suggesttion?

Best regard
Henry Chua

[Piergiacomo Perini]

Hi Henry,

"find" and "rm" it's the right way.
Pay attention from what directory the search it's starting ;-).

regards
pg

[Pete Randall]

Unless you can know in advance where the file is located, you really have no other choice than to use find. If you can narrow the scope of your search (i.e. "find /var/tmp" rather than "find /var") it would help reduce the amount of time required.


Pete

[Ralph Grothe]

If the inode can't be found in the buffer cache this can take some time, especially if you are doing a deep find.
So as suggested narrow find's entry path.
On Linux systems there's a tool called locate which usually is faster than find.
But it requires to have a current locate database built before (can be done by cron nightly).

[James R. Ferguson · Accepted solution]

Hi Henry:

You don't provide any details, but here is one crucial way to maintain performance:

If you are deleting more than just a very few files, pipe the 'find' output to 'xargs' to remove the files instead of using '-exec'.

Instead of:

# find /path -xdev -type f -mtime +1 -exec rm {} \;

...use:

# find /path -type f -mtime +1 | xargs rm

...This will collect groups of filenames as multiple arguments to 'rm' thereby spawning one process to remove many files at once instead of spawning one process to remove one file at a time --- a very expensive thing to do!

Regards!

...JRF...

[Henry Chua]

Thanks guys,

was wondering, if I were to use a C program to do this will there be any issue? Beside using scripting is there a faster option.. speed is imperative in this operation, so got to squeeze every ounce of juice from my box.. =)

Best regards
Henry

[James R. Ferguson]

Hi (again) Henry:

Relative to your last question of using a C program, without seeing your script, I still suspect that your performance issue is the 'exec' of multiple 'rm' processes as I delineated above.

C (or Perl) would allow you to call 'unlink(2)' which is the underlying function of 'rm'. Hence, for instance, using a Perl script and the "File::Find" module would eliminate spawning separate 'rm' processes, *BUT*, again, I still suspect that your performance issue is the 'exec' of multiple 'rm' processes as I delineated above.

Too, without seeing your whole script, it is difficult to postulate a better solution.

Regards!

...JRF...

[spex]

Hi Henry,

On my systems, I have functionality akin to the Linux 'locate' command, in which a search query is carried out by grepping a database of filenames.

To accomplish this, I have implemented the following cronjob:

10 01 * * * find / -local -print | compress - > /var/opt/locate/loc.db.Z # Refresh locate db

Then, to use 'locate':

# alias locate
locate='zcat /var/opt/locate/loc.db.Z | grep'
# locate '^/usr/.*/hostname$'
/usr/bin/hostname

Obviously, there are several drawbacks to this approach:

1) The database reflects the actual layout of filesystems only immediately after the database is created.

2) The database stores only filenames, not filetypes, permissions, modification time, etc. However, these other attributes could easily be added.

3) The database file can be altered by anyone with appropriate permission.

Because of these drawbacks (especially #1), I would only use 'locate' interactively (i.e. not in a script). But in this capacity, it works very well.

PCS

[Bill Hassell]

As mentioned, you must limit the scope of your find. Doing a:

find /

is the worst possible way to locate a file!!! Not only does it interfere with every other program, it will search a mounted CDROM and NFS mounted directories. You don't want that at all. You need to limit the search to the likely location of the file. For instance, only a root user can create files in directories like /usr/bin or /etc so there is no reason to search for user files in directories where they can't write a new file.

Perhaps if you can describe how the file gets created, we can suggest a much faster way to locate it. If you are looking for large files, there has been a LOT of comments here about why this is not a good idea for disk management.