topic Re: Continuously archiving directory contents in Operating System - HP-UX

Continuously archiving directory contents

Dermot Beirne — Fri, 22 Feb 2002 10:21:28 GMT

Hi,
I am wondering if there is a clever more efficient method to do what i am doing now. I have directories that are continuously growing with thousands of small files being created at a rate of over 100 every 5 minutes! Obviously this is a headache to deal with, and results in fbackup taking hours to run as some directories on this machine have over 1 million files in them by now. Anyway, I was going to tar the directories, delete the directory with rm -r and then recreate it again, but this would mean that files being created would fail for the time between directory deletion and recreation. Next I was going to run a recursive "for i in `ls -1, etc" in the directory to move the files to a tmp location and then delete them. The tmp location would then be archived and deleted itself. The problem with this is that i have to check the size of each file before moving it to prevent partial copies. Again with the number of files involved, and the number of directories this needs to be done with, the overhead would be too much. I am sure there is a much more clever way of archiving files to keep the qty down to take the pressure off fbackup. I am also sure that loads of sys admins must have to deal with this and someone has figured out how. I want to avoid "find" as they are too heavy on the processor, there could be 20 running at once on this box. Any help would be greatly appreciated.

Dermot.

Re: Continuously archiving directory contents

Tom Geudens — Fri, 22 Feb 2002 11:44:52 GMT

Hi,
Wow, that looks like fun :-).
I use the "for loop" you described. However, on my system I am sure that all but the most recent file are "complete". Using ls -rt1 makes it very easy to exclude this last file.

It may not be the solution you are looking for, but if your situation is simular, it would be a very easy one.

Regards,
Tom Geudens

Re: Continuously archiving directory contents

Carlos Fernandez Riera — Fri, 22 Feb 2002 12:47:03 GMT

Take the reverse way :

1- create several directories:

mkdir 0 1 2 3 4 5 6 7 8 ....

Now use symbolic links:

#! /usr/bin/ksh

let maxd=9
let dir=0

while [ 1 ]
do

let dir=$dir%$maxd

mv YOURDIR YOUROPENDIR
ln -s $dir YOURDIR
sleep 600
done

-----

In this way you will change of dir each 10 minutes. You will find some benefits:

1- open files remains open
2- shorter directories structures
3- Now you can test, backup, and remove dir by dir.

:-))

Re: Continuously archiving directory contents

Duncan Edmonstone — Fri, 22 Feb 2002 14:34:01 GMT

What about something like this, which would start moving the oldest files to a new location first, always checking whether a process has the file open...

olddir=/abc
newdir=/xyz
cd $olddir
ls -rt | while read f
do
open=`fuser $f 2>/dev/null`
if [ "$open" = "" ] ; then
mv $f $newdir
else
echo "file $f is held open by processes ${open}"
fi
done

This is untested, so you might want to create some test stuff first before using...
Also it won't work if there are sub-directories in the directory being archived

HTH

Duncan

Re: Continuously archiving directory contents

Dermot Beirne — Mon, 25 Feb 2002 14:26:34 GMT

Thanks all for your help. The script you have looks good Duncan, and I can modify it to move recursively through subdirectories. I believe that the reliability of fuser is questionable, but lsof seems to make up for this. I'll test extensively, and in our environment an errors will be detected *very* quickly :)

Dermot

Re: Continuously archiving directory contents

Duncan Edmonstone — Mon, 25 Feb 2002 18:18:38 GMT

Thanks for the points Dermot - you just pushed me onto my next hat! Wizard eh? that takes me back to very distant days of MUD!

So where do I get my FOD ;o)

Cheers

Duncan

Re: Continuously archiving directory contents

Steven Gillard_2 — Mon, 25 Feb 2002 18:30:57 GMT

I got around a similar problem once by doing some hard-link manipulation:

ln dirname dirname.
mkdir newdirname
# now force dirname to refer to the new directory...
ln -f newdirname dirname
unlink newdirname

This results in the existing directory being moved aside without affecting processes with files already open or processes that wish to create new files, because dirname always exists. A fresh new directory is now in place and you can do what you like with the old directory.

This may or may not work in your case, depending on your application, but it was quite effective in my case. I had the script running every 5 minutes which you could tweak depending on the rate at which new files are created.

Regards,
Steve