topic Re: check duplicate string in Operating System - HP-UX

check duplicate string

nash11 — Thu, 22 Jun 2006 20:50:17 GMT

I have a file , the content is :
aaa
bbb
ccc
ddd
eee
fff
ggg
aaa

the string "aaa" is duplicate , if I want to check any other duplicate string in the file , what can I do , can anyone help me ? thx

Re: check duplicate string

A. Clay Stephenson — Thu, 22 Jun 2006 20:59:25 GMT

sort < myfile | uniq -d

This will output any duplicate lines. Man uniq for details.

Re: check duplicate string

Hein van den Heuvel — Thu, 22 Jun 2006 21:31:56 GMT

sort and uniq are the right tools, as explained.

For small files I like using perl (or awk) associative array. For example:

# perl -ne "print if $x{$_}++" x.txt

Or

# perl -ne "print if 1==$x{$_}++" x.txt

Or

perl -ne "if (1==$x{$_}++) { print "duplicate: $_" } " x.txt

Cheers,
Hein.

Re: check duplicate string

Peter Nikitka — Fri, 23 Jun 2006 05:08:19 GMT

Hi,

you can even the the number of times (double, triple, ...) , a duplicate string occurs:

sort FILE | uniq -dc

mfG Peter

Re: check duplicate string

Ninad_1 — Fri, 23 Jun 2006 05:40:05 GMT

You can use
sort filename | uniq -d
to just list the entries which are duplicate (This will not list entries which occur only once)
sort filename | uniq or simply sort -u filename to get unique entries (This will omit the duplicate entries while displaying)
sort filename | uniq -dc | grep -v " 1 "
This will display entries which are duplicate with the count of their occurances.

Regards,
Ninad

Re: check duplicate string

Hein van den Heuvel — Fri, 23 Jun 2006 05:48:09 GMT

Hah... perl/awk can do that also:

awk '{a[$0]++} END {for (k in a){v=a[k]; if (v>1) print k, v}}' x

awk '{array[$0]++} END {for (key in array){val=array[key]; if (val>1) print key "=" val}}' x

:-)

Hein.

Re: check duplicate string

f. halili — Fri, 23 Jun 2006 07:42:30 GMT

# sort filename | uniq -d

cheers,
f. halili