Continuously archiving directory contents

[Dermot Beirne]

Hi,
I am wondering if there is a clever more efficient method to do what i am doing now. I have directories that are continuously growing with thousands of small files being created at a rate of over 100 every 5 minutes! Obviously this is a headache to deal with, and results in fbackup taking hours to run as some directories on this machine have over 1 million files in them by now. Anyway, I was going to tar the directories, delete the directory with rm -r and then recreate it again, but this would mean that files being created would fail for the time between directory deletion and recreation. Next I was going to run a recursive "for i in `ls -1, etc" in the directory to move the files to a tmp location and then delete them. The tmp location would then be archived and deleted itself. The problem with this is that i have to check the size of each file before moving it to prevent partial copies. Again with the number of files involved, and the number of directories this needs to be done with, the overhead would be too much. I am sure there is a much more clever way of archiving files to keep the qty down to take the pressure off fbackup. I am also sure that loads of sys admins must have to deal with this and someone has figured out how. I want to avoid "find" as they are too heavy on the processor, there could be 20 running at once on this box. Any help would be greatly appreciated.

Dermot.

[Tom Geudens]

Hi,
Wow, that looks like fun :-).
I use the "for loop" you described. However, on my system I am sure that all but the most recent file are "complete". Using ls -rt1 makes it very easy to exclude this last file.

It may not be the solution you are looking for, but if your situation is simular, it would be a very easy one.

Regards,
Tom Geudens

[Carlos Fernandez Riera]

Take the reverse way :


1- create several directories:

mkdir 0 1 2 3 4 5 6 7 8 ....


Now use symbolic links:

#! /usr/bin/ksh

let maxd=9
let dir=0

while [ 1 ]
do

let dir=$dir%$maxd

mv YOURDIR YOUROPENDIR
ln -s $dir YOURDIR
sleep 600
done

-----


In this way you will change of dir each 10 minutes. You will find some benefits:

1- open files remains open
2- shorter directories structures
3- Now you can test, backup, and remove dir by dir.


:-))

[Duncan Edmonstone]

What about something like this, which would start moving the oldest files to a new location first, always checking whether a process has the file open...

olddir=/abc
newdir=/xyz
cd $olddir
ls -rt | while read f
do
open=`fuser $f 2>/dev/null`
if [ "$open" = "" ] ; then
mv $f $newdir
else
echo "file $f is held open by processes ${open}"
fi
done

This is untested, so you might want to create some test stuff first before using...
Also it won't work if there are sub-directories in the directory being archived

HTH

Duncan

[Dermot Beirne]

Thanks all for your help. The script you have looks good Duncan, and I can modify it to move recursively through subdirectories. I believe that the reliability of fuser is questionable, but lsof seems to make up for this. I'll test extensively, and in our environment an errors will be detected *very* quickly :)

Dermot

[Duncan Edmonstone]

Thanks for the points Dermot - you just pushed me onto my next hat! Wizard eh? that takes me back to very distant days of MUD!

So where do I get my FOD ;o)

Cheers

Duncan

[Steven Gillard_2]

I got around a similar problem once by doing some hard-link manipulation:

ln dirname dirname.
mkdir newdirname
# now force dirname to refer to the new directory...
ln -f newdirname dirname
unlink newdirname

This results in the existing directory being moved aside without affecting processes with files already open or processes that wish to create new files, because dirname always exists. A fresh new directory is now in place and you can do what you like with the old directory.

This may or may not work in your case, depending on your application, but it was quite effective in my case. I had the script running every 5 minutes which you could tweak depending on the rate at which new files are created.

Regards,
Steve