Deleting massive directories quickly

[Dermot Beirne]

Hi,
I have two directories on a HP9000 running HPUX10.20.
These directories have 400000 and 700000 files in each of them respectively. I need to remove both of these. I started this last night, but the rm can only delete about 500 files a minute. These that the rm would be running over 20 hours to remove the directory with 700K files, and probably 15hours for the other one. Can anyone tell me of a quicker way to "zap" a directory than this!
Regards,
Dermot.

[Andreas Voss]

Hi,

if these diretories are extra mounted (/etc/fstab) i would umount, recreate with newfs ... and mount back the directory.

Regards

[Dermot Beirne]

Thanks Andreas,
However, these directories are not seperate filesystems, so I cannot use your suggestion. There are several other directories the same filesystem as these that I do not want to remove.

Dermot

[Robin Wakefield]

Hi Dermot,

If you're only using one process to do this, perhaps you could try running up a number of shells, cd'ing to the directory, and run the rm command on "blocks" of subdirs, e.g.

rm -r '[a-eA-E]*'
rm -r '[f-jF-J]*'
...
etc..., one per shell. This may not be practical of course, as I don't know the directory structure.

Rgds, Robin.

[Peter Kloetgen]

Hi Dermot,

if the other directories, which you still are smaller, do a backup of them and then follow the procedure of Andreas, create a new filesystem and make a restore of the needed data.

Allways stay on the bright side of life!

Peter

[Carlos Fernandez Riera]

Yes, it is easer and faster create a new lvol/fs and copy what you need, instead try to remove individual files.


You can try also with find and xargs, i:e:


find dir1 -mtime +30 | xargs rm.

It will run a rm with multiple files instead multiples rm of one file.

[Kenny Chau]

Well, I may suggest you to zip all the files into 1 file and delete that file.

Man gzip for information.

Hope this helps.
Kenny.

[Ian Dennison_1]

How about pulling a swifty?

Instead of 'delete', how about making them 'unavailable' by renaming the directory and recreating it as an empty area?

Nett effect: the files are no longer visible and you can delete them in slow time, while basking in the admiration of your Users that you managed to achieve this so quickly.

Share and Enjoy! Ian

[Steven Gillard_2]

How exactly are you removing the files, ie how are you using the rm command? Following on from Ian's suggestion I think the fastest way is to move aside the directories then run rm -r as it will be the quickest way.

If the directories are still in use, try some hardlink manipulation:

ln dirname dirname.old
mkdir dirname.new
# now force dirname to refer to the new directory...
ln -f dirname.new dirname
unlink dirname.new
# then remove the old dir with its contents...
rm -r dirname.old

Of course you're still going to be constrained by the speed of your hardware and the load thats being placed on it by other applications.

Regards,
Steve

[Dermot Beirne]

Your suggestions all have their own merits (including the swifty). But the machine is already extremely busy, and running rm's and find's are adding to the problem.
If I map the filesystem onto another server which is idle, and let that delete the files (all week if necessary) will there be any additional load on the box that actually has the files on it, or will it just ignore the deleting and work away itself.

Just a thought.
Dermot

[Dermot Beirne]

To answer my own previous question, and potentially as a tip for anyone else, mapping the offending directory onto another idle backup machine, and letting it do the deleting seems to be doing the trick.

Thanks for your help.
Dermot

[pap]

Hi Dermot,

You can do following things on your machine.

1. create a logical volume with name "junk"
2. Mount that volume on a directory /junk
3. Say you want to remove two directories A and B.
4. now give following commands

mv A /junk
mv B /junk

5. now you can remove the logical volume using lvremove.

I think this is the most convenient method for your case.

Thanks,

-Piyush

[Sanjay_6]

Hi Dermot,

Try "unlink" command. Do "manb unlink" for more info.

Hope this helps.

Regds

[Carlos Fernandez Riera]

Dermont:

can you explain further what you have done?

I cant understand...

If the filesystem is still mounted on the busy machine, mapping it from other box, dont free first box of manage filesystem. So, IMMO, it should be even worst.

TIA