topic Re: Script Help in Operating System - HP-UX

Script Help

Cem Tugrul — Thu, 02 Dec 2004 10:28:13 GMT

Hi forum,

Let's say i have a directory which includes thousand of files and i want to
compare each file's contents with the others bye one one and try to find out repeated
files(contents-records)

Help....

Re: Script Help

Steven E. Protter — Thu, 02 Dec 2004 10:32:02 GMT

The command is probably diff

diff file1 file2

You can build a script to read file lists and create diff output.

Do you need help setting up such a looping script?

SEP

Re: Script Help

Rodney Hills — Thu, 02 Dec 2004 10:33:03 GMT

I might do the following-

cksum * | sort

This would run checksum on all the files then sort by checksum value. Those files that were the same would sort together with the same checksum value.

HTH

-- Rod Hills

Re: Script Help

Fred Ruffet — Thu, 02 Dec 2004 10:35:26 GMT

Do you mean you want to suppress duplicate files or duplicate lines across different files.

Case 1 correponds to what SEP says (diff solution).

In case 2, you could cat all files through sort and uniq commands and get one file with unrepeated records.

Regards,

Fred

Re: Script Help

Ivajlo Yanakiev — Thu, 02 Dec 2004 10:36:10 GMT

you need loop:

for i in `ls`

do
for n in `ls`
do
diff $i $n >> /tmp/whatever
done
done

Re: Script Help

Cem Tugrul — Thu, 02 Dec 2004 10:54:17 GMT

Hi forum,
Thank's all answers...
Yes,i need help setting up such a looping script urgently...
Please help...

and Fred i need the contents(records) of all
files to compare and try to find out Ohh
these are the same files...
But my files names are different so maybe
best approach is files size...

Re: Script Help

Rodney Hills — Thu, 02 Dec 2004 11:03:19 GMT

If you are looking for files that are the same, what about my "cksum" solution?

It would be better then checking file size.

The "diff" solution others have given are to show how the files are different.

Maybe a little more explaination on what you have and why you are looking for "sameness".

-- Rod Hills

Re: Script Help

H.Merijn Brand (procura — Thu, 02 Dec 2004 11:13:28 GMT

Would my answer in this thread be the start to your solution?

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=749983

You can extend it to report duplicates like:

use Digest::MD5 qw( md5_hex );
use Digest::SHA1 qw( sha1_hex );
use File::Find;
my %arr;
find (sub {
-f or return;
local $/;
open my $p, "< $_" or die "$_: $!\n";
my $f = <$p>;
my $sum = md5_hex ($f) . sha1_hex ($f);
if (exists $arr{$sum}) {
print "File $File::Find::name is the same as file $arr{$sum}\n";
# unlink $_;
return;
}
$arr{$sum} = $File::Find::name;
}, ".");

Enjoy, Have FUN! H.Merijn

Re: Script Help

Fred Ruffet — Thu, 02 Dec 2004 11:16:02 GMT

I think Rodney's solution is very good. Using diff between every combination of two files will make you parse each file a huge number of time, whereas cksum once on each file and work on a ckecksum file would only arse once each file.
It should look like this :
cksum * > cksum.tmp
sort cksum.tmp > cksum.out
Then you can look at cksum.out. If two following lines have the same checksum it is the same file.

Regards,

Fred

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 01:30:53 GMT

Hi,

Let's say i have a directory as below;

-rw------- 1 cemt bsp 6 Dec 3 08:17 a.txt
-rw------- 1 cemt bsp 6 Dec 3 08:17 b.txt
-rw------- 1 cemt bsp 6 Dec 3 08:18 c.txt
-rw------- 1 cemt bsp 9 Dec 3 08:22 d.txt
-rw------- 1 cemt bsp 6 Dec 3 08:22 e.txt

Now i try to find out which files are the same???if you are a magician you can easily
say the file "a.txt" and "c.txt" are the same file!!!
Why;
Before cat these 5 files we can easily ignore the file "d.txt" because it's size is different than the others so;
Let's cat each file;

$ cat a.txt
11111
$ cat b.txt
22222
$ cat c.txt
11111
$ cat e.txt
33333

And we decided "a.txt" and "c.txt" are the same(repeated file)....is it clear???

Now i have more than 2000 files and try to find out repeated files in a directory?

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 02:14:53 GMT

Hi Procura,
also thank's for yourlink but i do not have any experience on perl.
How can i solve this problem with basic ux
scripting?(if possible)
Greetings,

Re: Script Help

H.Merijn Brand (procura — Fri, 03 Dec 2004 02:43:39 GMT

Did you try to put that snippet of perl in a file and execute it?

I've attached the (little modified) script, and show you an example:

lt09:/tmp 103 # find_dups.pl
File /tmp/mcop-merijn/Arts_PlayObjectFactory is the same as file /tmp/mcop-merijn/Arts_SoundServerV2
File /tmp/mcop-merijn/Arts_SoundServer is the same as file /tmp/mcop-merijn/Arts_SoundServerV2
File /tmp/mcop-merijn/Arts_SimpleSoundServer is the same as file /tmp/mcop-merijn/Arts_SoundServerV2
File /tmp/pics/zelfmoordactie.jpg is the same as file /tmp/blah/zelfmoordactie.jpg
File /tmp/pics/xmas.jpg is the same as file /tmp/blah/xmas.jpg

Enjoy, Have FUN! H.Merijn

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 03:02:45 GMT

Hi Procura,

i wanted to run it on my dir but faced
an error as ;
baan01:/users/rvs/usrdat/fave_org#perl find_dup.pl
Can't locate strict.pm in @INC (@INC contains: /opt/perl5/lib/5.00502/PA-RISC1.1 /opt/perl5/lib/5.00502 /opt/perl5/lib/site_perl/5.005/PA-RISC1.1 /opt/perl5/lib/site_perl/5.005 .) at find_dup.pl line 3.
BEGIN failed--compilation aborted at find_dup.pl line 3.

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 03:05:59 GMT

baan01:/#whereis perl
perl: /usr/contrib/bin/perl /opt/perl/bin/perl /opt/baf/bin/perl /opt/perl/man/man1/perl.1

do i have change perl path in your script?

Re: Script Help

H.Merijn Brand (procura — Fri, 03 Dec 2004 03:11:03 GMT

strict.pm SHOULD be there. There might be a small problem in the installation. Could you check

# which perl

and

# perl -v

and make sure /usr/contrib/bin is NOT somewhere in the front of your $PATH. /usr/contrib/bin usually contains old perls from HP.

but with the perl you have, it is very unlikely that you have the second Digest module installed.

Would you consider installing perl-5.8.5 from my site? It's more complete, includes a lot of modules (including the three I use in the example script) and is pretty recent.

I don't expect you to learn perl in a week or so, but once you uncover it's power, you will learn to love it fast!

Assuming you have a HP-UX 11.11 system, and no need for 64bit database connections just follow the instructions on http://mirrors.develooper.com/hpux/#Perl

My HP ITRC site pages can be found at (please use LA as primary choice):

USA Los Angeles http://mirrors.develooper.com/hpux/
SGP Singapore https://www.beepz.com/personal/merijn/
USA Chicago http://ww.hpux.ws/
NL Hoofddorp http://www.cmve.net/~merijn/

Enjoy, have FUN! H.Merijn

Re: Script Help

H.Merijn Brand (procura — Fri, 03 Dec 2004 03:31:55 GMT

Please move /usr/contrib/bin to the END of your $PATH. It'll save you a lot of annoyancies in the future.

It's old cruft!

Check /etc/PATH for it too

Enjoy, Have FUN! H.Merijn

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 03:59:55 GMT

$ which perl
/usr/contrib/bin/perl
$ perl -v

This is perl, version 5.005_02 built for PA-RISC1.1

Copyright 1987-1998, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5.0 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'. If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 04:36:41 GMT

hi Procura,
i have downloaded perl
perl-5.8.5-gcc-3.4.1-11.11-elf64
and my srv specs from (print_manifest);
Model: 9000/800/L1000-36
Main Memory: 2048 MB
Processors: 2
OS mode: 64 bit
Your system was installed with HP-UX version B.11.11.
How do i install perl 5.8.5?
Greetings,

Re: Script Help

H.Merijn Brand (procura — Fri, 03 Dec 2004 05:04:50 GMT

Note that that is a 64bit perl

# cd /opt
# bzip2 -d < /tmp/perl-5.8.5-gcc-3.4.1-11.11-elf64.tbz | tar xf -
# export PATH=/opt/perl64/bin:$PATH
# perl -v

Enjoy, Have FUN! H.Merijn

Re: Script Help

Cem Tugrul — Fri, 03 Dec 2004 05:28:55 GMT

oppsss,
this time i do not have bzip2:-(