topic comparing files - not sorted in Operating System - HP-UX

comparing files - not sorted

u856100 — Tue, 19 Nov 2002 08:48:22 GMT

hi,

I am trying to compare two large files (hence trying bdiff), but I think I am asking for a bit too much.

the files I want to compare are just a load of reference numbers, but the sorted nature of the file will mean I get an inaccurate picture, eg :

file 1 file 2
------ ------
a1 a1
b1 b1
c1 b2
d1 c1
e1 d1
e1

I want to find out which records do not exist in the other negating the order. So if I use bdiff, four descrepancies will be flagged, when I only want to know about the fact that b2 exists in one file and not the other ... is this possible, am I asking too much. The actual files have 3 Million entries.

thanks a lot
john

Re: comparing files - not sorted

Thierry Poels_1 — Tue, 19 Nov 2002 08:51:31 GMT

Hi,

3 million entries?
I would start thinking of loading this stuff into a database, create the necessary indexes and query the database for the required results.

good luck,
Thierry.

Re: comparing files - not sorted

H.Merijn Brand (procura — Tue, 19 Nov 2002 09:17:18 GMT

perl using a tied hash.

warning, not tested.

#!/usr/bin/perl

use strict;
use warnings;

use DB_File;
tie my %f1, "DB_File", "f1_tie";
@ARGV = ("f1");
while (<>) {
chomp;
$f1{$_}++;
}
@ARGV = ("f2");
while (<>) {
chomp;
if (exists $f1{$_}) {
print "= $_ $f1{$_}\n";
$f1{$_} = 0;
}
else {
print "> $_\n";
}
}
for(keys%f1) {
$f1{$_} or next;
print "< $_ $f1{$_}\n";
}
untie %f1;

Re: comparing files - not sorted

John Palmer — Tue, 19 Nov 2002 09:31:39 GMT

Hi,

If you do a unique sort of both files, 'comm' will tell you which records only appear in one file and not the other.

man comm

Regards,
John

Re: comparing files - not sorted

u856100 — Tue, 19 Nov 2002 11:05:04 GMT

Hi John,

I've just tried a quick test using

file1 file2
----- -----
a c
b d
c b
a
a

when I do $comm -12 file1 file2

it gives me: c

But obviously a,b,and c appear in both files

bit confused!

cheers
John

Re: comparing files - not sorted

John Palmer — Tue, 19 Nov 2002 11:27:57 GMT

As stated in the man page, both files should be sorted, yours are not!

If you sort your example files, then run comm -12, you'll get the correct answer (b and c, a isn't in your second file!).

Actually, from your original post, you should be using comm -23 to list records in file1 and not file2.

Regards,
John