Move duplicate files to hold directory

[syedar]

Hi all,

Please check the script,

The list of files is
ab.5890.20090311130443
ab.6065.20090311214416
ab.5890.20090311124512
ab.6065.20090311212449
ab.5891.20090311124512

#!/bin/ksh
for fls in `ls -1 /data/Syed/ab/ab*`
do
sqn=`echo $fls|cut -d"." -f2`
if [ `ls /data/Syed/ab/ab.$sqn.* | wc -l` -gt 1 ]
then
mv ab*.$sqn.* hold/
fi
done

The result is :
Moving: ab.5890.20090311124512 ab.5890.20090311130443
Moving: ab.5890.20090311124512 ab.5890.20090311130443
Moving: ab.6065.20090311212449 ab.6065.20090311214416
Moving: ab.6065.20090311212449 ab.6065.20090311214416

Just i need to check the files of same sequence and compare the size of same seqence files and move file which has small in size to hold directory.

[Mark McDonald_2]

ok, so you are already getting the files you need in the hold dir. Can you confirm that you want to leave the larger of the 2 files behind? AND only move the smaller to the hold dir?

An easy way would be to check just before you run the mv.

Hi all,

Please check the script,

The list of files is
ab.5890.20090311130443
ab.6065.20090311214416
ab.5890.20090311124512
ab.6065.20090311212449
ab.5891.20090311124512

#!/bin/ksh
for fls in `ls -1 /data/Syed/ab/ab*`
do
sqn=`echo $fls|cut -d"." -f2`
if [ `ls /data/Syed/ab/ab.$sqn.* | wc -l` -gt 1 ]
then

SMALLER='ls -l | grep -v total | awk '{ print $5 " " $9 }' | sort -rn | tail -1'

mv $SMALLER hold/
fi
done

[Mark McDonald_2]

ls -l puts a total at the top of my output, not sure if it will on yours? This results in a blank line after the awk. So I grep -v total to get rid of this. This will cause a problem if some of your file names have the work total in them.

So I awk out the size and filename
sort this based on size and reverse it so that the smallest file is output last, then tail -1 gives the last File.

NOTE: you will need a further awk '{ print $2 }' at the end like so:


SMALLER=$(ls -l | grep -v total | awk '{ print $5 " " $9 }' | sort -rn | tail -1 | awk '{ print $2}' )

[syedar]

Hi Mark,

If you see the result the output is repeatinf twice and first move works fine but on seconf move it is giving error that "file move error" because files already moved to hold directory.
ab.5890.20090311124512 ab.5890.20090311130443

when iam using your new added line
SMALLER=$(ls- l | grep -v total | awk '{ print $5 " " $9 }' | sort -rn | tail -1 | awk '{ print $2}' )

Message is coming:
Usage: mv [-f] [-i] [-e warn|force|ignore] f1 f2
mv [-f] [-i] [-e warn|force|ignore] f1 ... fn d1
mv [-f] [-i] [-e warn|force|ignore] d1 d2
Usage: mv [-f] [-i] [-e warn|force|ignore] f1 f2
mv [-f] [-i] [-e warn|force|ignore] f1 ... fn d1
mv [-f] [-i] [-e warn|force|ignore] d1 d2
Usage: mv [-f] [-i] [-e warn|force|ignore] f1 f2
mv [-f] [-i] [-e warn|force|ignore] f1 ... fn d1
mv [-f] [-i] [-e warn|force|ignore] d1 d2

and finally it is moving "Find_dupl_files.sh" script because it is small in size but i need to move the files from the list which is having same sequence and smaller in size mentioned in my previous mail.

[syedar]

Hi Mark,

This works fine, Thank you.

[syedar]

Thanks Mark.

[Mark McDonald_2]

How about some points then?