topic sort data into monthly directories in Operating System - Linux

sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 04:37:35 GMT

Hi everyone,

I have a file that lists all data in a directory ( about 2 years worth of data )

I need to search the list file and create a directory for each month and year ie

archive..
archive.outdata.0703

I have an idea how to do this however am stuggling getting the sytax.

I will then need to move the files in outdata to the relevant archive directory.

please any help will be much appreciated.

Thanks

Chris.

Re: sort data into monthly directories

SANTOSH S. MHASKAR — Wed, 11 Apr 2007 04:52:39 GMT

Hi,

Pl. clearify following,

1] I have a file that lists all data in a directory ( about 2 years worth of data )

What is the data format of the that 'lists all data in a directory', give sample.

2] I need to search the list file and create a directory for each month and year

list file is the same file that 'lists all data in a directory'

3] I will then need to move the files in outdata to the relevant archive directory.

What is 'outdata' ?

Re: sort data into monthly directories

Dennis Handly — Wed, 11 Apr 2007 04:55:40 GMT

archive..
archive.outdata.0703

Is this the contents of your data file? Or the resultant directory you want to create?

Is "archive.outdata.0703" is a directory with files you want to move?

>I will then need to move the files in outdata to the relevant archive directory.

Where is your list of files?
And you show some lines of your data file??

Re: sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 04:57:07 GMT

Hi,

1)What is the data format of the that 'lists all data in a directory', give sample.

the list files are produced from ls -l:

-rw-r--r-- 1 950 staff 43 05 Oct 2006 AST0000015
-rw-r--r-- 1 950 staff 43 17 Oct 18:42 AST0000016

2)list file is the same file that 'lists all data in a directory'

yes that is correct - the file that details every file to be moved to the newly created directory

3) What is 'outdata' ?

outdata is an example of the directory name where the files exist.

Thanks

Re: sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 05:11:02 GMT

archive..
archive.outdata.0703

Is this the contents of your data file? Or the resultant directory you want to create?

--> this will be the resulying directory - the contents file is called outdata

Is "archive.outdata.0703" is a directory with files you want to move?

--> all files with March date stamp will be moved to archive.outdata.0703 directory once created by the script

Thanks

I am working through an awk book to attempt the correct syntax however not found the answer yet.

Thanks for the help

Re: sort data into monthly directories

SANTOSH S. MHASKAR — Wed, 11 Apr 2007 06:27:20 GMT

Hi,

go to the dir where ur dir is located
execute following script

----------------------------
set -vx
Cur_Year=`date +'%Y'`
for File_cur_year in `ls -lt|awk 'length($8) == 5 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
for File_cur_year in `ls -lt|awk 'length($8) == 4 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
------------------------------------

Re: sort data into monthly directories

SANTOSH S. MHASKAR — Wed, 11 Apr 2007 06:36:14 GMT

Hi ,

Sorry one line to add in second for loop for
Cur_Year variable

----------------------
Cur_Year=`date +'%Y'`
for File_cur_year in `ls -lt|awk 'length($8) == 5 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
for File_cur_year in `ls -lt|awk 'length($8) == 4 {print $9}'`
Cur_Year=`ll ${File_cur_year}|awk 'length($8) == 4 {print $6`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done

Re: sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 07:44:27 GMT

ok thanks man,

I will make a start with this syntax as I still require the month to be displayed as a numeric ie archive.outdir.0703

this indicates 07 being the year and 03 being the month.

cheers

Chris

Re: sort data into monthly directories

Hein van den Heuvel — Wed, 11 Apr 2007 08:53:13 GMT

I would recommend not using a listing with pre-formatted ugly dates.
Use perl to 'glob' through the files;
'stat' the files found for the last modification time in seconds
'localtime' to translate to year and month
'sprintf' to nicely format
mkdir if needed
rename
presto.

See working example below
Now adapt to your eact needs.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

$cat move.pl
use strict;
use warnings;
my $source = shift or die "Please provide source path";
my $target = shift;
$target = "archive" unless $target;
my %seen;
while (<$source/*>) {
my ($m,$y) = (localtime((stat)[9]))[4,5];
my $yymm = sprintf ("%s.%02d%02d", $target, $y%100, ++$m);
my $old = $_;
s/^$source/$yymm/;
# print "$m $y $old $_\n";
if (!exists($seen{$yymm})) {
mkdir $yymm unless -d $yymm;
$seen{$yymm}++;
}
rename $old, $_;
}
#-----------------------------------
$ touch -t 0102030405 a
$ touch -t 0105030405 b
$ touch -t 0505030405 c
$ touch -t 0505030405 d
$ ls -l
total 0
-rw-rw-rw- 1 hein 513 0 Feb 3 2001 a
-rw-rw-rw- 1 hein 513 0 May 3 2001 b
-rw-rw-rw- 1 hein 513 0 May 3 2005 c
-rw-rw-rw- 1 hein 513 0 May 3 2005 d
$ perl move.pl x
2 101 archive.0102 archive.0102/a
5 101 archive.0105 archive.0105/b
5 105 archive.0505 archive.0505/c
5 105 archive.0505 archive.0505/d
$ ls -l arc*

archive.0102:
total 0
-rw-rw-rw- 1 hein 513 0 Feb 3 2001 a

archive.0105:
total 0
-rw-rw-rw- 1 hein 513 0 May 3 2001 b

archive.0505:
total 0
-rw-rw-rw- 1 hein 513 0 May 3 2005 c
-rw-rw-rw- 1 hein 513 0 May 3 2005 d

Re: sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 09:06:02 GMT

excellent thanks again my friend

Re: sort data into monthly directories

lawrenzo — Wed, 11 Apr 2007 09:33:53 GMT

Hein,

This doesn't work for me ....

can you please elabotate on the syntax as I am not familiar with perl ...

This will be a one time operation and I have a list of files as the directory where the data is being moved will need to have 3 months worth of data in as I have developed another script to dothis monthly from syntax you have previously supplied.

Thanks again for the help.

Chris

Re: sort data into monthly directories

James R. Ferguson — Wed, 11 Apr 2007 11:29:35 GMT

Hi Chris:

Copy Hein's script and add the interpreter (she-bang) at the very beginning just as you would do for a shell or 'awk' script:

#!/usr/bin/perl

Then, run it passing the path name of the directory you want to archive:

# ./move.pl /home/chris/oldstuff

The script globs (creates a list of files) in the source directory you pass as the first argument.

This file list is sucessively read and the 'mtime' (modification) timestamp in your localtime derived from a 'stat()' of each file in the list. You are only interested in the month and year and as you can see, the month is returned zero-relative and the year as the year-1900.

So as not to return an error for existing directory's, each year-month combination is tracked in a hash. If a year-month combination that hasn't already occured is encountered, a 'mkdir' is issued.

The 'rename()' system call is that which is used by the 'mv' command in a shell.

If you already have a file that contains the *names* of the files you want to archive, you could amend Hein's script to something like this variation:

#!/usr/bin/perl
use strict;
use warnings;
my $target = shift;
$target = "archive" unless $target;
my %seen;
while (<>) {
chomp;
my ($m,$y) = (localtime((stat)[9]))[4,5];
my $yymm = sprintf ("%s.%02d%02d", $target, $y%100, ++$m);
my $old = $_;
s/^/$yymm/;
if (!exists($seen{$yymm})) {
mkdir $yymm unless -d $yymm;
$seen{$yymm}++;
}
rename $old, $_;
}

In this case do:

# cd /path #...containing the files named with relative paths in 'myfile':

# ./move2.pl myfile

Regards!

...JRF...

Regards!

...JRF...

Re: sort data into monthly directories

lawrenzo — Thu, 12 Apr 2007 05:49:49 GMT

ok will give it a whirl ....

Thanks guys.

Chris.