Operating System - Linux
1748136 Members
3624 Online
108758 Solutions
New Discussion юеВ

sort data into monthly directories

 
SOLVED
Go to solution
lawrenzo
Trusted Contributor

sort data into monthly directories

Hi everyone,

I have a file that lists all data in a directory ( about 2 years worth of data )

I need to search the list file and create a directory for each month and year ie

archive..
archive.outdata.0703

I have an idea how to do this however am stuggling getting the sytax.

I will then need to move the files in outdata to the relevant archive directory.

please any help will be much appreciated.

Thanks

Chris.
hello
12 REPLIES 12
SANTOSH S. MHASKAR
Trusted Contributor

Re: sort data into monthly directories

Hi,

Pl. clearify following,

1] I have a file that lists all data in a directory ( about 2 years worth of data )

What is the data format of the that 'lists all data in a directory', give sample.

2] I need to search the list file and create a directory for each month and year

list file is the same file that 'lists all data in a directory'

3] I will then need to move the files in outdata to the relevant archive directory.

What is 'outdata' ?
Dennis Handly
Acclaimed Contributor

Re: sort data into monthly directories

archive..
archive.outdata.0703

Is this the contents of your data file? Or the resultant directory you want to create?

Is "archive.outdata.0703" is a directory with files you want to move?

>I will then need to move the files in outdata to the relevant archive directory.

Where is your list of files?
And you show some lines of your data file??
lawrenzo
Trusted Contributor

Re: sort data into monthly directories

Hi,

1)What is the data format of the that 'lists all data in a directory', give sample.

the list files are produced from ls -l:

-rw-r--r-- 1 950 staff 43 05 Oct 2006 AST0000015
-rw-r--r-- 1 950 staff 43 17 Oct 18:42 AST0000016

2)list file is the same file that 'lists all data in a directory'

yes that is correct - the file that details every file to be moved to the newly created directory

3) What is 'outdata' ?

outdata is an example of the directory name where the files exist.

Thanks
hello
lawrenzo
Trusted Contributor

Re: sort data into monthly directories

archive..
archive.outdata.0703

Is this the contents of your data file? Or the resultant directory you want to create?

--> this will be the resulying directory - the contents file is called outdata

Is "archive.outdata.0703" is a directory with files you want to move?

--> all files with March date stamp will be moved to archive.outdata.0703 directory once created by the script

Thanks

I am working through an awk book to attempt the correct syntax however not found the answer yet.

Thanks for the help
hello
SANTOSH S. MHASKAR
Trusted Contributor

Re: sort data into monthly directories

Hi,

go to the dir where ur dir is located
execute following script

----------------------------
set -vx
Cur_Year=`date +'%Y'`
for File_cur_year in `ls -lt|awk 'length($8) == 5 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
for File_cur_year in `ls -lt|awk 'length($8) == 4 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
------------------------------------
SANTOSH S. MHASKAR
Trusted Contributor

Re: sort data into monthly directories

Hi ,

Sorry one line to add in second for loop for
Cur_Year variable

----------------------
Cur_Year=`date +'%Y'`
for File_cur_year in `ls -lt|awk 'length($8) == 5 {print $9}'`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
for File_cur_year in `ls -lt|awk 'length($8) == 4 {print $9}'`
Cur_Year=`ll ${File_cur_year}|awk 'length($8) == 4 {print $6`
do
if [ ! -d ${File_cur_year} ]
then
File_cur_mon=`ll $File_cur_year|awk 'length($8) == 5 {print $6}'`
else
File_cur_mon=`ls -ld $File_cur_year|awk 'length($8) == 5 {print $6}'`
fi
if [ ! -d archive.outdata.${File_cur_year}${File_cur_mon} ]
then
mkdir archive.outdata.${Cur_Year}${File_cur_mon}
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
else
cp ${File_cur_year} archive.outdata.${Cur_Year}${File_cur_mon}
fi
done
lawrenzo
Trusted Contributor

Re: sort data into monthly directories

ok thanks man,

I will make a start with this syntax as I still require the month to be displayed as a numeric ie archive.outdir.0703

this indicates 07 being the year and 03 being the month.

cheers

Chris
hello
Hein van den Heuvel
Honored Contributor
Solution

Re: sort data into monthly directories

I would recommend not using a listing with pre-formatted ugly dates.
Use perl to 'glob' through the files;
'stat' the files found for the last modification time in seconds
'localtime' to translate to year and month
'sprintf' to nicely format
mkdir if needed
rename
presto.

See working example below
Now adapt to your eact needs.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting


$cat move.pl
use strict;
use warnings;
my $source = shift or die "Please provide source path";
my $target = shift;
$target = "archive" unless $target;
my %seen;
while (<$source/*>) {
my ($m,$y) = (localtime((stat)[9]))[4,5];
my $yymm = sprintf ("%s.%02d%02d", $target, $y%100, ++$m);
my $old = $_;
s/^$source/$yymm/;
# print "$m $y $old $_\n";
if (!exists($seen{$yymm})) {
mkdir $yymm unless -d $yymm;
$seen{$yymm}++;
}
rename $old, $_;
}
#-----------------------------------
$ touch -t 0102030405 a
$ touch -t 0105030405 b
$ touch -t 0505030405 c
$ touch -t 0505030405 d
$ ls -l
total 0
-rw-rw-rw- 1 hein 513 0 Feb 3 2001 a
-rw-rw-rw- 1 hein 513 0 May 3 2001 b
-rw-rw-rw- 1 hein 513 0 May 3 2005 c
-rw-rw-rw- 1 hein 513 0 May 3 2005 d
$ perl move.pl x
2 101 archive.0102 archive.0102/a
5 101 archive.0105 archive.0105/b
5 105 archive.0505 archive.0505/c
5 105 archive.0505 archive.0505/d
$ ls -l arc*

archive.0102:
total 0
-rw-rw-rw- 1 hein 513 0 Feb 3 2001 a

archive.0105:
total 0
-rw-rw-rw- 1 hein 513 0 May 3 2001 b

archive.0505:
total 0
-rw-rw-rw- 1 hein 513 0 May 3 2005 c
-rw-rw-rw- 1 hein 513 0 May 3 2005 d




lawrenzo
Trusted Contributor

Re: sort data into monthly directories

excellent thanks again my friend
hello