AWK: compare & add (or sed or perl or . . .)

[Robert A. Pierce]

Good day!

I am comparing two files, and want to compare the numbers in the second field.

The files have data like this:

Field Name (spaces) Numeral
>
AMASSET 1542
AMASSETADJ 1552
AMASSETCNV 0
<

Note: Not all field names are in both files, so there isn't a line-by-line relationship between the two.

I want to compare the files' fields and counts.
I couldn't get AWK to search on a variable so I ended up using grep in a shell script, like this:

> grep "^$FIELD[ ]" count*

which gave me output like this:

>
AMASSET
count_1:AMASSET 1542
count_2:AMASSET 1542
-----------------------------
AMASSETADJ
count_1:AMASSETADJ 1552
count_2:AMASSETADJ 1575
-----------------------------
AMASSETCNV
count_1:AMASSETCNV 0
count_2:AMASSETCNV 0
<

With this, I can do an eyeball-grep see that the two AMASSETADJ values are different, but with over 6000 lines my eyes get tired.

Is there an easier way to do this? I'd like to flag the comparasons that are different, or even only print those out.

Thanks,

Rob Pierce

[Juergen Tappe]

Hi Rob,
an elegant method here is the join command.

Try : join file1 file2
or : join -a 1 -a 2 file1 file2

you might want to sort the files to temp files before that.

The disadvantage here is that, if a value is only in one file, you cannot see in which.
Or I dint found a quick method to .....

[Juergen Tappe]

Another Idea:
use a shell script similar to :

fields=$(awk '{print $1}' file1 file2 | sort -u)

for i in $fields
do
f1=$(grep "^$i " file1 | awk '{print $2}')
f2=$(grep "^$i " file2 | awk '{print $2}')
f1=${f1:-missing}
f2=${f2:-missing}
echo $i $f1 $f2

[Juergen Tappe]

... sorry missed final line in the script:

add "done" at the end :)

[H.Merijn Brand (procura]

# perl -ne'/^(\w+)\s+(\d+)/ and$x{$1}{$ARGV}=$2}BEGIN{($f1,$f2)=@ARGV}END{for(sork keys%x){$x{$_}{$f1}==$x{$_}{$f2} and next;print"$_\n$f1:$_ $x{$_}{$f1}\n$f2::$_ $x{$_}{$f2}\n"' count_1 count_2

should work. not tested (only prints if the numbers are not equal).

Enjoy, Have FUN! H.Merijn

[Robert A. Pierce]

Juergen,

The join command is much better than my script. Thanks!

The second script prints the differences in fields, but not the differences in values when both files have the same field.

I should be able to use awk to compare $2 and $3 in the "join"ed file. That makes it much simpler.

Thanks!

[Robert A. Pierce]

Procura,

I got this:

Can't find string terminator "'" anywhere before EOF at -e line 1.

Ah ... should that be "sort keys" instead of "sork keys?"

Let me try it again . . . nope.

hmm . . .

[H.Merijn Brand (procura]

Yes. sork->sort, and missing }

lt09:/home/merijn 106 > cat count_1
blah 1
foo 2
baz 3
lt09:/home/merijn 107 > cat count_2
blah 2
foo 2
baz 3
lt09:/home/merijn 108 > perl -ne'/^(\w+)\s+(\d+)/ and$x{$1}{$ARGV}=$2}BEGIN{($f1,$f2)=@ARGV}END{for(sort keys%x){$x{$_}{$f1}==$x{$_}{$f2} and next;print"$_\n$f1:$_ $x{$_}{$f1}\n$f2::$_ $x{$_}{$f2}\n"}' count_1 count_2
blah
count_1:blah 1
blah 2
lt09:/home/merijn 109 >

Enjoy, Have FUN! H.Merijn [ Who used the format you showed in your own example as output ]

[Francisco J. Soler · Accepted solution]

Hi Robert,

Check this awk script:

cat count1 count2 | awk '{
if ($1 in a) {
if (a[$1]!=$2)
printf("%s %d\n%s %d\n\n",$1,a[$1],$1,$2)
}
else
a[$1]=$2
}'

This script only puts in the stdout the fields that have different values in the second field.

Hope this helps.

Frank.

[Robert A. Pierce]

Thanks very much for all your help and answers.

I'm using Francisco's awk script and it works very well.

Rob

[Hein van den Heuvel]

Here is a solution with a slightly different perl approach: less perlish, more plain programming.

Create the following script file:

$file = shift;
open (F1, "<$file") or die "Failed to open first file: $file.";
while () {
$fields{$1} = $2 if (/(\w+)\s+(\w+)/);
}
$file = shift;
open (F2, "<$file") or die "Failed to open second file: $file.";
while () {
if (/(\w+)\s+(\w+)/) {
$f1 = $fields{$1};
print "$1 1:$f1 2:$2\n" if (defined $f1 && $f1 != $2);
}
}


Now:
perl compare-script file1 file2

With that framework is will be easy to enhance if need be.
- like printing field name from file2 if no corresponding found in file1
- detect duplicates or missing in file2 (hint: delete keys when matched first, report remaining keys when done).

Enjoy,
Hein.

[Rachel Watts]

I am searching for the author of The Terrible Beast From the Deep. This is a long shot but are you this author?

[Robert A. Pierce]

Rachel,

Sorry, but no, I'm not.

Happy new year!

Rob

[Robert A. Pierce]

Thanks again, all.

I used Francisco's awk script for the issue then at hand, but found the other script examples very helpful for a better understanding of TMTOWTDI.

Happy New Year, all!