- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- Re: How to delete large number of files
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 04:29 AM
тАО11-30-2005 04:29 AM
How to delete large number of files
We have a directory that holds around 1 million files. What is the fastest way to delete all those files short of rebuilding the filesystem the directory is housed in.
I used
"find . -type f -print | xargs /bin/rm"
and it took 20 hours (on a D380).
There must be a faster way...
Thanks
Rayb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 04:34 AM
тАО11-30-2005 04:34 AM
Re: How to delete large number of files
Well....the quickest way IS going to be destroying/rebuilding the LV.
The problem is the dir size. With 1 million files *every* lookup and inode mod is going to take a long time - especially on a K-class.
Do you *really* need that many files in a single dir?
If you do then I'd suggest that the dir be an FS & LV unto itself. Then you can knock it down & rebuild it w/o touching anything else.
My 2 cents,
Jeff
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 04:34 AM
тАО11-30-2005 04:34 AM
Re: How to delete large number of files
rm -R
mkdir
adjust owner/group/permissions as required.
Not sure if this would be faster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 04:38 AM
тАО11-30-2005 04:38 AM
Re: How to delete large number of files
Well, as you said, the fastest way is to rebuild the filesystem.
Consider: If you are deleting the vast majority of files, it may be faster to backup what you want to keep; rebuild the filesystem; and reload what you kept.
If you can specify whole subdirectories in the filesystem, I'd certainly use:
# rm -rf /mountpoint/dir1 /mountpoint/dir2 ...
If you need some subdirectories retained, but not others, use a combination of the recursive 'rm' above and a 'find -type f' *limited* to those subdirectories where you want to remove only the files.
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 04:52 AM
тАО11-30-2005 04:52 AM
Re: How to delete large number of files
We might be able to rearrange the question so that the time required doesn't matter very much:
1) rename the existing directory
2) create a new directory with the name of the old
3) start your rm process on the renamed directory in the background - nohup'ed.
The idea is that you have a fresh new directory available for use while the old one is being cleaned up and the cleanup time is not nearly so critical.
Still, Plan A should be divide and conquer so that you never have that many files in a directory. In general, filesystems are poor substitutes for databases.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО11-30-2005 05:23 AM
тАО11-30-2005 05:23 AM
Re: How to delete large number of files
To sum it up, you need a plan that does not leave you 1 million or even 100,000 files in the same directory.
Thats what subdirectories are for.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-07-2005 01:53 AM
тАО12-07-2005 01:53 AM
Re: How to delete large number of files
I fully agree that it's too big but it's a legacy and I, personally, can't do much about it at this moment.
One good thing, this directory is the only thing on that file system.
Does anyone know of a faster way to back it up/ restore it than using "ftio"?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-07-2005 02:01 AM
тАО12-07-2005 02:01 AM
Re: How to delete large number of files
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-07-2005 02:12 AM
тАО12-07-2005 02:12 AM
Re: How to delete large number of files
It fails on an excessive file list.
That's why we use ftio.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО12-07-2005 02:13 AM
тАО12-07-2005 02:13 AM