topic Re: Finding duplicate filenames in Operating System - HP-UX

Finding duplicate filenames

Preet Dhillon — Mon, 15 Oct 2001 09:53:14 GMT

I'd like to find all files with the same names in a given filesystem. The difficulty is that I don't know what the filenames are - could be anything. Also, the files could live in dozens of different directories. Is there a way of listing all duplicate files with full paths to where they are ?
Many thanks in advance.

Re: Finding duplicate filenames

G. Vrijhoeven — Mon, 15 Oct 2001 10:23:54 GMT

Hi Preet:

to list duplicate names:

find . type f | awk -F / '{ print $NF }' | sort | uniq -d

hope this will help

Re: Finding duplicate filenames

Sridhar Bhaskarla — Mon, 15 Oct 2001 10:24:01 GMT

Preet,

Give this a try

#!/usr/bin/ksh

if [ $# -ne 1 ]
then
echo "$0: full path to the directory/filesystem"
exit 1
fi

DIR=$1

if [ ! -d $DIR ]
then
echo "$DIR no such directory"
exit 1
fi

if [ -f duplicates ]
then
rm duplicates
fi

find $DIR -type f > list$$
awk '{FS="/";print $NF}' list$$ |sort |uniq -d >> files$$
for FILE in `cat files$$`
do
grep $FILE list$$ >> tmp$$
for ENTRY in `cat tmp$$`
do
ONE=`echo $ENTRY|awk '{FS="/";print $NF}'`
if [ $ONE = $FILE ]
then
echo $ENTRY >> duplicates
fi
done
done

rm tmp$$ files$$ list$$
echo "Check the file "duplicates" in the current dir"

-Sri

Re: Finding duplicate filenames

Wim Rombauts — Mon, 15 Oct 2001 10:26:31 GMT

I think you best start with a "find" and put the output in a textfile. Then you can work on the textfile to find duplicate filesnames.

The find command looks like this :
find -print > filenames.txt

With 'sed', you can covert this file to a list of filenames without path and the patch itself on the same line. Then sort this file.

Re: Finding duplicate filenames

Sridhar Bhaskarla — Mon, 15 Oct 2001 10:29:04 GMT

Preet,

You can cut and paste my script. It gives the full paths to the duplicate files in a file called "duplicates" in the current directory.

-Sri

Re: Finding duplicate filenames

harry d brown jr — Mon, 15 Oct 2001 10:32:59 GMT

This will "GRAB" the file name from the path, put the file name first, and then the directory after it. That way you get what you waht, duplicate files in different directories.

find /var -type f|sed "s/$^\/.*\/$$.*$$/\2 \1/"|sort >filenames.out

Re: Finding duplicate filenames

Stefan Farrelly — Mon, 15 Oct 2001 10:51:35 GMT

And the winner is....Vrijhoeven

His answer handles spaces in file and director names (very important!) but needs some massaging to be complete;

1. cd ; find . -print > /tmp/tempfile
2. for i in $(find . -type f | awk -F / '{print $NF}'|sort|uniq -d)
do
grep "/${i}" /tmp/tempfile
done

Tested it, works great.

Re: Finding duplicate filenames

Frank Slootweg — Mon, 15 Oct 2001 10:59:20 GMT

Maybe this is a start. I think this does what you want. It lists *all* files, but it lists the duplicate ones first.

$ cat ../doit
#! /usr/bin/sh

find . -type f |
{
while read filename
do
echo `dirname $filename` `basename $filename`
done
} | sort -k 2,2

$ ls -R
dir1 dir2 dir3

./dir1:
a c d

./dir2:
a b e

./dir3:
b c f

$ ../doit
./dir1 a
./dir2 a
./dir2 b
./dir3 b
./dir1 c
./dir3 c
./dir1 d
./dir2 e
./dir3 f
:;

sort -k 2,2

Re: Finding duplicate filenames

Sridhar Bhaskarla — Mon, 15 Oct 2001 11:02:20 GMT

Not to point out but...

Winner's script doesn't work in this scenario.
My directory structure is

.
./test1
./test1/test
./test1/sri
./test2
./test2/test
./test2/new
./test3
./test3/test
./test3/sri
./file

-Sri

Re: Finding duplicate filenames

Frank Slootweg — Mon, 15 Oct 2001 11:13:05 GMT

OOPS! Typos!
Please ignore the stuff after "./dir3 f", i.e. the silly ":;" and the standalone "sort -k 2,2 ".

Re: Finding duplicate filenames

Stefan Farrelly — Mon, 15 Oct 2001 11:59:38 GMT

oops, change the last grep to;

grep "${i}$" /tmp/tempfile

This will find the filenames at the end of the line. Works with the above test1,2,3 directory structure now.

Re: Finding duplicate filenames

Robin Wakefield — Mon, 15 Oct 2001 12:22:31 GMT

Hi,

This method avoids having to run find twice:

===========================================
find . -type f | while read file ; do
echo $(dirname "$file")'\t'$(basename "$file")
done |
sort -t" " -k 2 - > /tmp/listfiles
uniq -f 1 -u /tmp/listfiles |
join -t" " -v 2 -1 2 -2 2 -o 2.1,2.2 - /tmp/listfiles |
sed 's+ +/+'
=============================================

Both join and sort commands have the tab character between the double quotes.

Rgds, Robin.

Re: Finding duplicate filenames

Stefan Farrelly — Mon, 15 Oct 2001 12:27:08 GMT

Hey Robin,

you sure your script works with spaces in a directory and filename ? usually dirname and basename dont work with spaces in names.

Re: Finding duplicate filenames

Robin Wakefield — Mon, 15 Oct 2001 12:36:57 GMT

Hi Stefan,

Seems OK, as long as it's quoted.

# basename /tmp/a b c
# NOTHING !!

# dirname "/tmp/a b c"
/tmp
# basename "/tmp/a b c"
a b c

I tried a few variations, and it seemed to behave itself.

Cheers, Robin.

Re: Finding duplicate filenames

Robin Wakefield — Mon, 15 Oct 2001 12:56:34 GMT

...and there's a tab between the 1st pair of "+" characters in the final sed...Robin

Re: Finding duplicate filenames

Stefan Farrelly — Mon, 15 Oct 2001 13:04:10 GMT

Thanks for checking that Robin. OK, now we have 2 answers which do the business.

Preet - dont forget to assign points.

Re: Finding duplicate filenames

harry d brown jr — Wed, 17 Oct 2001 10:59:01 GMT

Here is a perl script (attachment) I found that will actually find DUPLICATE files, and not just duplicate file names.

credits:

http://www.geocities.com/fcheck2000/download.html