topic Re: comm command in Operating System - HP-UX

comm command

amonamon — Wed, 21 Mar 2007 06:40:51 GMT

well I am not sure is only comm command convinient for this problem but I tryes with it..

fileA -

11
33
44
55
77

fileB -

22
34
45
55
56
66
77
778
888
999

output:

11
33
44

so print to output line from fileA if it is not in fileB..

thnaks a lot..

Re: comm command

john korterman — Wed, 21 Mar 2007 06:46:46 GMT

Hi,

try;
$ comm -23 fileA fileB

regards,
John K.

Re: comm command

Dennis Handly — Wed, 21 Mar 2007 06:55:30 GMT

Both fileA and fileB need to be sorted for comm(1).

Re: comm command

Peter Nikitka — Wed, 21 Mar 2007 07:06:48 GMT

Hi,

a sorted file in the sight of 'comm' is a collated sort. If you have a numerical sorted input like
9
88
100
it's complete out of order in lexical (=comm) sense.

mfG Peter

Re: comm command

amonamon — Wed, 21 Mar 2007 08:21:47 GMT

It works as I tried also with comm -23 and this can work for small file but I have 1 000 000 lines in files and it has some problems..I will test it again and let U know..

Thanks a lot!!!

Re: comm command

Peter Godron — Wed, 21 Mar 2007 08:35:26 GMT

Hi,
seems you have a duplicate at:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1110742

Have you tried:
grep -vf fileB fileA

How big are both of your files ?

Re: comm command

Dennis Handly — Wed, 21 Mar 2007 16:53:22 GMT

>Peter N: a sorted file in the sight of 'comm' is a collated sort.

Yes. The above fragments were sorted by ASCII. To check you can use:
$ sort -c fileA; echo $?
$ sort -c fileB; echo $?

>Peter G: Have you tried: grep -vf fileB fileA
>How big are both of your files?

With a million lines, that may be too big.
Using sort & comm instead, may depend on how often these files change.

Re: comm command

Bob E Campbell — Wed, 21 Mar 2007 17:22:35 GMT

I am not sure what the performance of this solution would be, but I have to believe it would beat the double sorted comm(1) solution:

# grep -v -f fileB fileA
11
33
44

Re: comm command

Bob E Campbell — Wed, 21 Mar 2007 17:45:25 GMT

Ooops, sorry Peter and Dennis. I need to get new bifocals. I generated a file with numbers from 1 to 4000000 and then copied it and deleted a few lines:

# time comm -13 sortedListA sortedListB
123456
15
3999966

real 0.8
user 0.7
sys 0.0

# time comm -13 bigListA bigListB
15
123456
3999966

real 0.7
user 0.7
sys 0.0

# time grep -v -f bigListA bigListB
grep: not enough memory

real 9:57.2
user 9:53.2
sys 0.6

Looks to me as if the two files need to have the same sort order, that grep can't stomach comparing two really big files, and 8 gig of RAM just isn't what it used to be.

Bob

Re: comm command

Dennis Handly — Wed, 21 Mar 2007 19:56:14 GMT

>Bob: Looks to me as if the two files need to have the same sort order, that grep can't stomach comparing two really big files

sort doesn't use a bubble sort. ;-)
So it maybe takes n1 * log(n1) + n2 * log(n2)
Doing a grep requires comparing every record in fileA with everyone in fileB (perhaps 50% if it matches, 100% if it doesn't).

comm only needs to compare with records up to the match.