topic Re: shell script problem.. in Operating System - HP-UX

shell script problem..

amonamon — Tue, 02 Oct 2007 04:25:43 GMT

Hello can anyone give fast and good solution on this:
I have 2 files:
fileA:

629122
629145
629111
629000
629999
629787
629444

fileB:
123569
789999
629122
629145
777777
629111
000000
629999
629787
629444
989898
777777

and result should be :

629000

becouse it is the only line that is in fileA and not in fileB

any help..I tryed something with comm..but..no use..

Re: shell script problem..

Oviwan — Tue, 02 Oct 2007 04:28:54 GMT

hey

check the diff command:
$diff fileA fileB

Regards

Re: shell script problem..

MarkSyder — Tue, 02 Oct 2007 04:32:43 GMT

One of those "scrap of paper" moments - I haven't tried this but think it will work:

for i in `cat fileA`
do
if grep $i fileB
then
:
else
echo $i
done

Mark Syder (like the drink but spelt different)

Re: shell script problem..

john korterman — Tue, 02 Oct 2007 04:36:27 GMT

Hi,

try this first:
$ sort fileA >./fileA.s
$ sort fileB >./fileB.s

$ comm -23 fileA.s fileB.s
629000

regards,
John K.

Re: shell script problem..

amonamon — Tue, 02 Oct 2007 04:38:41 GMT

noo...
Problem is becouse file B is very big and also fileA

files are not sorted..
so most of content from fileA is in fileB but there are 10-20 lines that are in fileA that fileB does not have..

I think I got it..
sort -b fileA > fileA.sort
sort -b fileB > fileB.sort

comm -23 fileA.sort fileB.sort

Re: shell script problem..

Ian Lochray — Tue, 02 Oct 2007 04:48:13 GMT

grep -v -f fileb filea

Re: shell script problem..

MarkSyder — Tue, 02 Oct 2007 04:52:56 GMT

Ian,

Your solution will show every line that doesn't match. If one line matches and the other (say) 3000 don't, it will show him those 3000 lines.

Mark

Re: shell script problem..

Dennis Handly — Tue, 02 Oct 2007 05:39:31 GMT

>can anyone give fast and good solution on this:

Fast to run? Or fast to implement?

>Mark: I haven't tried this but think it will work:

Using grep will have terrible performance for large files. Using that cat(1) will also tokenize each line, if there is more than one field that would fail.

John has the right solution with sort and comm. If there are duplicates, you may or may want to use sort -u.

>noo... Problem is because file B is very big and also fileA

Were you complaining about the sort, sort, comm solution?

>Mark: Ian, Your solution will show every line that doesn't match.

Isn't that what amonamon wants? It gives 629000. It will have all the lines of fileA that are not in fileB.

Re: shell script problem..

Dennis Handly — Tue, 02 Oct 2007 05:50:00 GMT

If fileA and fileB are about 2000 records, I get:
grep:
real 0m0.02s user 0m0.03s sys 0m0.01s

sort, sort, comm:
real 0m0.07s user 0m0.01s sys 0m0.01s

With 30,000 they break even and comm is better.

Re: shell script problem..

MarkSyder — Tue, 02 Oct 2007 05:50:06 GMT

Dennis,

I am assuming that Ian's solution would involve a loop like my earlier suggestion. The first iteration of the loop would do grep -v 123569 fileA. This would output every line in fileA that did not contain 123569. Similarly, the second iteration would output every line that did not contain 789999 etc.

As Ian has written it, amonamon would actually be searching for the string fileB in fileA.

Mark

Re: shell script problem..

Dennis Handly — Tue, 02 Oct 2007 05:59:44 GMT

>Mark: I am assuming that Ian's solution would involve a loop like my earlier suggestion.

Any loop would be IN grep. That's what grep -f does. Looks at the file fileB for each string.

>The first iteration of the loop would do grep -v 123569 fileA.

No. grep looks at each record of fileA and if it matches any line of fileB, it isn't printed. Perhaps you have your gedanken for-loops interchanged. :-)

>As Ian has written it, amonamon would actually be searching for the string fileB in fileA.

That's not what "grep -v -f file" does. Did you try it?

Re: shell script problem..

Ernesto Cappello — Tue, 02 Oct 2007 07:06:45 GMT

Hi Amonamon, this is the command:

diff A B | grep "<" | awk '{print $2}'

to satisfy your request.

Best regards.
Ernesto

Re: shell script problem..

MarkSyder — Tue, 02 Oct 2007 07:20:55 GMT

Dennis,

No, I missed the -f!

I've never used it and have not tried it but will bear it in mind for future use.

Mark

Re: shell script problem..

Sandman! — Tue, 02 Oct 2007 10:48:45 GMT

> any help..I tryed something with comm..but..no use..

Did you sort the files before filtering your criteria thru comm(1) i.e.

# sort fileA > sfA
# sort fileB > sfB
# comm -23 sfA sfB
629000

Re: shell script problem..

blah2blah — Tue, 02 Oct 2007 10:55:22 GMT

a faster alogrithm would be to use awk or perl to read in file b put everything in an array or hash, the read file a and check to see if entry is in the hash if it isn't print it.

something like this. maybe someone that knows perl can correct my mistakes.

open(FILEHANDLE,"
while (){
@a[chop()]=1;
}

open(FILEHANDLE,"
while (){
if @a[chop()] print;
}

Re: shell script problem..

Sandman! — Tue, 02 Oct 2007 12:29:37 GMT

never mind my post...John Korterman beat me to it by a mile :)