data archive

[maliaka]

I need to keep 6 months of data and archive the 7th month everytime the data grows older than 6 months.
10 points for the best solution!

[Dennis Handly]

Are you talking about a file, a few files, a directory tree, a filesystem, database or a whole system?
How much data, in Mb? And archive to what?

[maliaka]

Bunch of directories and files under one parent directory. I'm using a tool to do the archive but I have to enter the month so I just need help with the logic to know that the data are older than 6 months.

[James R. Ferguson]

Hi:

This will find files older than 180-days and copy them to another directory. Directory heirarchies are duplicated as necessary:

# cd /srcpath
# find . -type f -mtime +180|cpio -pudlmv /dstpath
# find . -type f -mtime +180 -exec rm {} +

The second 'find' removes the files that met the criteria for copy.

Regards!

...JRF...

[Dennis Handly]

>I just need help with the logic to know that the data is older than 6 months.

On a file by file basis?
JRF's find command can list just the filenames:
find . -type f -mtime +180

Note: That is any file not modified in 180 days.

[maliaka]

Sorry guys but I'm having trouble explaining what I need so let me try again.
Let's say the timestamp of the parent directory has Dec., the logic will automatically run the archive tool and pass june as the month to archive.
I'm just having difficulty creating that logic.

[Dennis Handly · Accepted solution]

>Let's say the timestamp of the parent directory has Dec., the logic will automatically run the archive tool and pass june as the month to archive.

It's clear now. :-)

Basically you can use fancy smancy calendar scripts like Clay's caljd.sh:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1157244

Or you can use a simple shell case to look up the month string and "subtract" 6 months:
mon=$(ll -d $directory | awk '{print $6}')
case $mon in
Jan) mon6="July";;
Feb) mon6="August";;
...
Dec) mon6="June";;
esac

archive_tool $mon6

[James R. Ferguson]

Hi (again):

Here are two rigorous variations of my original theme:

# cat .arch1.sh
#!/usr/bin/sh
typeset SRCDIR=/tmp/dummydir1 # change
typeset DSTDIR=/tmp/dummydir2 # change
typeset MYLIST=/tmp/mylist.$$
trap 'rm ${MYLIST}' EXIT

cd ${SRCDIR} || exit 1
find . -type f -mtime +180 -print > ${MYLIST}
cat ${MYLIST} | cpio -pudvm ${DSTDIR}
rm $(<${MYLIST}) 2> /dev/null
exit 0

# cat .arch2.sh
#!/usr/bin/sh
typeset SRCDIR=/tmp/dummydir1
typeset DSTDIR=/tmp/dummydir2
typeset MYLIST=/tmp/mylist.$$
trap 'rm ${MYLIST}' EXIT

cd ${SRCDIR} || exit 1
find . -type f -mtime +180 -print > ${MYLIST}

while read FILE
do
DIR=$(dirname ${FILE})
mkdir -p ${DSTDIR}/${DIR}
cp -p ${FILE} ${DSTDIR}/${DIR}
done < ${MYLIST}

rm $(<${MYLIST}) 2> /dev/null
exit 0

Regards!

...JRF...

[maliaka]

Dennis,

You got it!
Now, it gets a little complicated with the year.

From your first example:

Jan) mon6="July";;

How can I get the previous year?

[Dennis Handly]

>it gets a little complicated with the year. How can I get the previous year?

That's an issue. Also, by using ll on a directory, you get the time of the last modification. So if nobody is fiddling in it, the time should be the time the last file was created.
Also note, if the time on the file is over 6 months old, it is in the Mon Day Year format. So are you going to run this tool on directories that are less than 6 months?

You could look for the ":" and determine it is the current year.
Otherwise you could use the following on old files to get the year:
mon=$(ll -d $directory | awk '{print $6}')
year=$(ll -d $directory | awk '{print $8}')
case $mon in
Jan) mon6="July"; (( year -= 1));;
...
Jun) mon6="December"; (( year -= 1));;
Jul) mon6="January";;
...
Dec) mon6="June";;
esac

[James R. Ferguson]

Hi (again):

OK, so you want to examine the last modification timestamp of a directory and deduce the date 6-months ago, adjusted for year boundries. Use Perl:

# cat ./sixago
#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
my $dir = shift;
die "$0: directory expected\n" unless -d $dir;
my $moddate = ( stat($dir) )[9];
my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst )
= localtime($moddate);
my $modtime = timelocal( 0, 0, 0, 15, $mon, $year ) - ( 180 * 86400 );
print join " ", ( split / /, localtime($modtime) )[ 1, 4 ], "\n";
1;

...run as:

# ./sixago /directory

For example:

# touch 02290800 /tmp/mydir
# ./sixago /tmp/mydir
Aug 2007

# touch /tmp/mydir
# ./sixago /tmp/mydir
Jun 2008

This works by fetching the last modification time (in seconds) for the directory. This is converted into a month-day-year-time format. The month and year are used and the day of the month set to 15. This modified date is then converted back to Epoch seconds and 180-days worth of seconds (one day = 86400 seconds) are subtracted. The new date in seconds is then converted to a month-day-year-time format and the local month name and year are extracted and returned.

Regards!

...JRF...

[James R. Ferguson]

Hi (again):

...and to use the Perl script and return the date to your shell script, simply do:

...

./archive_tool $(./sixago /somedirectory)

or:

ARCHDATE=$(./sixago /somedirectory)

Regards!

...JRF...