System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Selecting Unique Values from many List

 
SOLVED
Go to solution
pareshan
Regular Advisor

Selecting Unique Values from many List

I have a question which is giving me a serious problem



I have like 19 different list which contains the name of the server but what I need is just unique ones.



First thing I need to do is just make a unique list within the list itself

i.e. delete anything that is repeated inside the list

like for example



in list1

i have

a

b

c

d

e

a

c

f

g

h





so i just need is a b c d e f g h i mean unique values which is easier anyone can do it i guess.

For the first list its ok



Like that i have 19 liist and i have to do that the same thing which is ok not that hard



the problem is now,



there should not be anything repeating in any one of those list

i.e. if something is there in the first list means that should not be in 2nd list and so on. so that means

for the first list its ok i dont need to do anything as it is after deleting the duplicates



second list I have to compare with the first list and delete if anything from the first list



third list i have to compare with first and second list and anything present in first or second list I have to delete from third list and so on upto 19 list



Plz can anyone suggest me how can i do that



let me know if anything confusing



just i want is not repeated in any one of those 19 list i hope thats pretty much clear



Thanks In advance

its serious problem im having
12 REPLIES
James R. Ferguson
Acclaimed Contributor

Re: Selecting Unique Values from many List

Hi:

If you don't mind sorting and you want one list when done:

# sort -u file1 file2 file3 ...

or if the filenames are similar:

# sort -u f[0-9]*

Regards!

...JRF...
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

Thaks James but looks like you dint get what my problem is

I dont want one list i want all the 19 list but

there should not be anything repeating in any one of those list

i.e. if something is there in the first list means that should not be in 2nd list and so on. so that means

for the first list its ok i dont need to do anything as it is after deleting the duplicates



second list I have to compare with the first list and delete if anything from the first list



third list i have to compare with first and second list and anything present in first or second list I have to delete from third list and so on upto 19 list

Thanks
OldSchool
Honored Contributor

Re: Selecting Unique Values from many List

so, for each of the 19 files, you want to find only the unique entries *within* that file....

and then for each of those nineteen, you want to remove any entries which appear in the prior "x" files?????

such that when working with the 16th file, any entry in 16 which appears in 1-15 should be ommitted?????

"sort -u file1 > file1s" will get you the unique values and needs to be done for each file....

using those results, you can do:

grep -v -f file1s -f file2s -f file3s ....
-f file15s file16u > file16su

would remove any entries occuring the sorted/unique results of 1-15 from the 16, leaving only those unique to 16.
Steven Schweda
Honored Contributor

Re: Selecting Unique Values from many List

> [...] looks like you dint get what my
> problem is

He's not alone.



Perhaps you could show us what you want to
get, instead of trying to describe it.




Perhaps more blank lines would help.
James R. Ferguson
Acclaimed Contributor
Solution

Re: Selecting Unique Values from many List

Hi (again):

OK, let's see if this gives you what you want:

# cat ./reduce
#!/usr/bin/perl
use strict;
use warnings;
my %seen;
while (<>) {
print;
chomp;
$seen{$_}++;
if (eof) {
print "------\n";
last;
}
}
while (<>) {
chomp;
print "$_\n" unless exists $seen{$_};
$seen{$_}++;
print "------\n" if eof;
}

...Run as:

# ./reduce f1 f2 f3 ...

For example, given:

# cat f1
a
b
c
d
e
f

# cat f2
a
c
f
g
h
i

# cat f3
a
f
h
i
k
l
m

# ./reduce f1 f2 f3
a
b
c
d
e
f
------
g
h
i
------
k
l
m
------

Regards!

...JRF...
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

Steven,

Oldschool got my problem, thats what I wanted to do.
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

James Thanks alot
it prefectly works but how can I find which output is for which file

I know its in the order I provided in the command line but

----is there any way I can have name of the file in the beginning----------

Thanks alot
really Great job
James R. Ferguson
Acclaimed Contributor

Re: Selecting Unique Values from many List

HI (again):

Use this to trace the file names:

# #!/usr/bin/perl
use strict;
use warnings;
my %seen;
while (<>) {
print;
chomp;
$seen{$_}++;
if (eof) {
print "[ $ARGV ] ------\n";
last;
}
}
while (<>) {
chomp;
print "$_\n" unless exists $seen{$_};
$seen{$_}++;
print "[ $ARGV ] ------\n" if eof;
}

# ./reduce f1 f2 f3
a
b
c
d
e
f
[ f1 ] ------
g
h
i
[ f2 ] ------
k
l
m
[ f3 ] ------

...using the data I used in my previous post.

Regards!

...JRF...
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

It dint work for my list. I tried with just 2 and dint work so i dint try with many

cat uniq_from_many_list.pl
#!/usr/bin/perl
use strict;
use warnings;

my %seen;
while (<>) {
print;
chomp;
$seen{$_}++;
if (eof) {
print "------\n";
last;
}
}

while (<>) {
chomp;
print "$_\n" unless exists $seen{$_};
$seen{$_}++;
print "------\n" if eof;

}


cat USAGE3s
astro
bhpnbi2c
bhprtg13
bhprtg13
uhprtg6
uhprtg6
uhprtg6

cat MAFGGDBs
astro
benji
bhpmaf13
bhpmaf14
bhpmaf15
bhpmafg7
bhpmafg8
dhpmaf17
dhpmafg1
dhpmafg1
dhpmafg2
dhpmafg4
dhpmafg6
uhpmafg5


uniq_from_many_list.pl USAGE3s MAFGGDBs
astro
bhpnbi2c
bhprtg13
bhprtg13
uhprtg6
uhprtg6
uhprtg6
------
astro
benji
bhpmaf13
bhpmaf14
bhpmaf15
bhpmafg7
bhpmafg8
dhpmaf17
dhpmafg1
dhpmafg1
dhpmafg2
dhpmafg4
dhpmafg6
uhpmafg5
------



Anything need to be changed so that it will work ?
James R. Ferguson
Acclaimed Contributor

Re: Selecting Unique Values from many List

Hi (again):

If you have leading and or trailing whitespace in any line, the original code would consider this significant. This would account for the failure with your data. we can remdedy this:

# cat ./reduce
#!/usr/bin/perl
use strict;
use warnings;
my %seen;
while (<>) {
print;
chomp;
s/^\s*//; s/\s*$//;
$seen{$_}++;
if (eof) {
print "[ $ARGV ] ------\n";
last;
}
}
while (<>) {
chomp;
s/^\s*//; s/\s*$//;
print "$_\n" unless exists $seen{$_};
$seen{$_}++;
print "[ $ARGV ] ------\n" if eof;
}
1;

Regards!

...JRF...
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

Great!!!!!!!
Thank You So Much

ITs working like i wanted
pareshan
Regular Advisor

Re: Selecting Unique Values from many List

I have found the solution