topic Re: A script in Operating System - HP-UX

A script

ivy1234 — Tue, 22 Mar 2011 04:55:08 GMT

I have a file , the file as below ,

aa,bb,cc
aa,cc,dd
cc,dd,ee
ff,zz
dd,aa

the file have many lines, and the content may be duplicated and separated by "," sign . Now if I want to erase some contents in the file

1) if the content is duplicate , then output 1 time
2) the result should be in 1 line.

so my desired output is

aa
bb
cc
dd
ee
ff
zz

can advise what can i do ? thx

Re: A script

Raj D. — Tue, 22 Mar 2011 05:26:27 GMT

ivy,

check this out:

$ cat file | tr "," "\n" | uniq -u

Hth,
Raj.

Re: A script

ivy1234 — Tue, 22 Mar 2011 06:24:14 GMT

thx ,

but it do not handle duplicate case , that mean the output is deplicate ,

can advise if I want if the data is duplicated then do not output the same data , what can i do ? thx

Re: A script

ivy1234 — Tue, 22 Mar 2011 06:30:54 GMT

thx

the |uniq -u seems not work in this case.

Re: A script

Raj D. — Tue, 22 Mar 2011 08:15:29 GMT

ivy,
You can use uniq -c and cut the numeric first field,

$ cat file | tr "," "\n" | uniq -c | cut -c 1-2

I cant check it now, as don't hv system now.
Hth,
Raj.

Re: A script

Steven Schweda — Tue, 22 Mar 2011 08:18:37 GMT

man uniq
man sort

alp$ < 1474162.txt tr ',' '\n' | sort | uniq
aa
bb
cc
dd
ee
ff
zz
alp$

Try it first without the "| uniq".

Re: A script

Hein van den Heuvel — Tue, 22 Mar 2011 11:26:48 GMT

For 'uniq' to work the stream has to be sorted first

Man...
"DESCRIPTION
Discard all but one of successive identical lines from INPUT"

Here the solution with "tr | sort | uniq" probably works just fine.

For modest dataset you may also want to play with PERL to allow for more tricky splitting, parsing, counting and printing.

In this simple example we can set up an array value for each word found and at the end ( eskimo kiss: }{ :-) print all the keys thus established

$ perl -lne '$x{$_}=1 for split /,/ } { print for (sort keys %x) ' x.txt
aa
bb
cc
dd
ee
ff
zz

fwiw,
Hein

Re: A script

Raj D. — Tue, 22 Mar 2011 18:17:09 GMT

Ivy,
Here you go with awk,
# cat file | tr "," "\n" | awk '!x[$0]++'

aa
bb
cc
dd
ee
ff
zz
#

Enjoy, Have fun! Remember to assign points to all posts,
Raj.

Re: A script

Mel Burslan — Tue, 22 Mar 2011 18:41:18 GMT

Raj is still missing the point. uniq only senses consecutive lines which were duplicates

aa
bb
aa
bb
bb

ran thru uniq, will generate:

aa
bb
aa
bb

not

aa
bb

the one liner should be something like this:

cat file | tr "," "\n" | sort | uniq

Hope this helps

Re: A script

Steven Schweda — Tue, 22 Mar 2011 19:01:23 GMT

> cat file | tr "," "\n" | sort | uniq

Geez. Why didn't _I_ think of that. No,
wait...

And my version lacked the (much hated) "cat".
And, when picoseconds count, I figure that
'x' should be faster than "x" -- no looking
for dollar signs in 'x'.

Re: A script

Dennis Handly — Wed, 23 Mar 2011 04:10:34 GMT

>Steven: ... | sort | uniq

You can optimize this by using "sort -u".

Re: A script

Raj D. — Wed, 23 Mar 2011 04:57:47 GMT

Mel, thanks good thing leaned abt uniq,
, uniq only senses consecutive lines which were duplicates. sometime I used to wonder why uniq not working properly, now it make sense, t u.

Re: A script

Arturo Galbiati — Wed, 23 Mar 2011 07:18:30 GMT

cat file | tr ',' '\n'|sort -u
HTH,
Art

Re: A script

Raj D. — Wed, 23 Mar 2011 07:47:21 GMT

# cat file|tr "," "\n" | awk '!x[$0]++' #Enjoy!.