Operating System - Linux
1748157 Members
4093 Online
108758 Solutions
New Discussion юеВ

Re: file with duplication ignor anything where there is a duplicate.

 
SOLVED
Go to solution
Sandman!
Honored Contributor

Re: file with duplication ignor anything where there is a duplicate.

If the file has mixed-case names and you want to keep it that way, then the awk script I posted earlier will suffice. In case you want to ignore case of the names modify the awk construct as:

# awk '{x[tolower($1)]++}END{for(i in x) if(x[i]==1) print i}' file

~cheers
rmueller58
Valued Contributor

Re: file with duplication ignor anything where there is a duplicate.

It's in the awk vault.. Thanks Sandman, I can see the others are useful, I can find places for them as well.

Merry Christmas all.
spex
Honored Contributor

Re: file with duplication ignor anything where there is a duplicate.

Hi Rex,

This can be accomplished by commands alone:

$ sort file | uniq -c | grep '1 ' | cut -c6-

Merry Christmas!

PCS
rmueller58
Valued Contributor

Re: file with duplication ignor anything where there is a duplicate.

Spex, I tried that it leaves the dups in place.. Need to have none of the records that have duplicates..

James R. Ferguson
Acclaimed Contributor

Re: file with duplication ignor anything where there is a duplicate.

Hi Rex:

OK, silly me, I thought that your file contained only records with the listed fields.

Consider this file:

aanderson line-1
abergman line-2
abergman line-3
aboell line-4
aboell line-5
abone line-6
abridwell line-7
abridwell line-8
aburks line-9
achowdhury line-10

Now use:

# cat ./report
#!/usr/bin/perl
use strict;
use warnings;
my $file = shift;
my %names;
my @fields;
open (FH, "<", $file) or die "Can't open '$file': $!\n";
while () {
@fields = split;
$names{$fields[0]}++;
}
seek( FH, 0, 0);
while () {
@fields = split;
print if $names{$fields[0]} == 1;
}
1;

...thus:

# ./report file
aanderson line-1
abone line-6
aburks line-9
achowdhury line-10

...Perl counts the first field as zero whereas 'awk' would count it as one.

Regards!

...JRF...
rmueller58
Valued Contributor

Re: file with duplication ignor anything where there is a duplicate.

Thats the deal Jim! Thanks AGAIN!..