cancel
Showing results for 
Search instead for 
Did you mean: 

Shell script dont work with awk.

SOLVED
Go to solution
Ulf Furn
Regular Advisor

Shell script dont work with awk.

Hi everybody. I hope that some one can help me with this,
I trying to identify identical ip addresses by comparing to file, this is not a problem.

The problem is that i cant seem to read the values back to a file.

This is the code I use!

IDENTICAL=(`cat IdenticalIp.txt`)
IDENTRIES=${#IDENTICAL[@]}

while [ $index3 -lt $IDENTRIES ]
do
awk 'match($0, "${IDENTICAL[index3]}") == 0 {print$0}' $OVIP >> toBeExcluded.txt
echo "${IDENTICAL[index3]}"
index3=$((index3+1))
done

The awk command works when I run it from the command line and use a ip address insted of my array. it seems that awk dont handle the array variable correct! HELP!
Best regards //Stefan
15 REPLIES
Hein van den Heuvel
Honored Contributor

Re: Shell script dont work with awk.

The shell variable ${index3} never makes it into your awk construct.
The value for index3 within awk will be undefined / 0.

You can fix that with more a more tricky argument line, doing the substitution before getting into awk, or you can just pass the variable explicitly into awk:

$ test="aap"
$ awk -vtest=$test 'BEGIN {print test}'
aap

-vindex3=$index3

? and how is that array getting into awk?


But why not just in awk and do everything that needs to be done right there?
Just suck that indentical file into an awk array in its 'BEGIN' section.

Something like:
$ cat > tmp.txt
aap
noot
mies
$ awk 'BEGIN { while (getline lines[i++] < "tmp.txt"); print lines[2],lines[1],lines[0]}'

mies noot aap


For you that would like like:

$ awk 'BEGIN { while (getline IDENTICAL[i++] < "IdenticalIp.tx");} { .. main code body .. }' $OVIP >> toBeExcluded.txt

Please provide a few useful example lines of the two input file and a corresponding output file if more help is needed.

Good luck!
Hein.
James R. Ferguson
Acclaimed Contributor

Re: Shell script dont work with awk.

Hi Stefan:

Pass the array element to 'awk':

# awk -v element=${IDENTICAL[index3]} 'match($0, element) == 0 {print $0}'

Regards!

...JRF...
Dennis Handly
Acclaimed Contributor

Re: Shell script dont work with awk.

>IDENTICAL=(`cat IdenticalIp.txt`)

No need for () nor evil cats:
IDENTICAL=$(< IdenticalIp.txt)

Unless you have made IDENTICAL an array, it isn't:
set -A IDENTICAL $(< IdenticalIp.txt)

>index3=$((index3+1))

This can be done as:
(( index3 += 1 ))

Also instead of reading everything into an array, you can use a while to read out of IdenticalIp.txt:
while read IDENTICAL; do
awk -v IDENTICAL=$IDENTICAL 'match($0, IDENTICAL) == 0 {print$0}' $OVIP >> toBeExcluded.txt
echo "${IDENTICAL}"
done < IdenticalIp.txt

Also there is no need to use match, you should be able to use !~ for matching.
Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

Hi and thanks for your responses. But it dont work any way.The awk command still dont compare and exclude the identical input.

For input in the loop, its pass 21 addresses.
So when I pass this to awk it outputs it 21 times. Summery 441 lines.
If I only let the loop write to the output file ones. it exclude one address and leaves the rest.

IDENTICAL=(`cat IdenticalIp.txt`) #Containes identical ipaddresses from earlier output.
IDENTRIES=${#IDENTICAL[@]} #Array containes 21 elements

echo "$IDENTRIES"
while [ $index3 -lt $IDENTRIES ]
do
awk -v element=${IDENTICAL[index3]} 'match($0, element) == 0 {print$0}' $OVIP > toBeExcluded.txt
#echo "${IDENTICAL[index3]}"
index3=$((index3+1))
done

# The file $OVIP is the file to compare the addresses. toBeExcluded.txt should only after processeing, contain those addresses that are to be excluded. The identical adressed should not be in the output file.
Hein van den Heuvel
Honored Contributor

Re: Shell script dont work with awk.

Do you really want the lines from IdenticalIp.txt to be treated as regular expression, such as 'match' will?

It will make 1.1.1.1 match 121.191.3.4
Is that desirable or acceptable?

If it is, why not just use:

# grep -v -f IdenticalIp.txt $OVIP > toBeExcluded.txt

If it is not acceptable, then you may need to use something like

substr($0,1,length(element) == element { ...}
of course you could do all that in shell code.

And if you do not check the filter file as you read it, then it should be very certain that it is 'clean'. No trailing spaces, or empty lines to force unexpected matches.

So far this remains an n x m algoritme as coded. (21*21=441).
Is that acceptable at production time?
Is either of the files expected to be much larger than the other?

If you read the filter into an AWK, or PERL associative array as per my first reply, and then use the IP from the second file as a key to attempt an array lookup, then it becomes and n + m algorithm (21 + 21=42).
It will only start up perl/awk once.
Much more efficient! ... if that matters at all

Your example is cute, but too smart for me.
I think it would help get better replies if you provide a few simple test lines from the input files and desired output. Just paste them here, and if spaces are important, also paste them into a .TXT file to be attached.

Hope this helps,
Hein.
Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

Well, I thought so!
The file that are read in to the array and then walk trough the loop, contains only ip addresses. But when I looking att this more and more, am not sure.

How can "match" do a match between 1.1.1.1 and 121.192.4.5 if I use a dedicated ip address as regexp? I am missing something here. The loop dont just loop trough the 21 entries in the array, it loops trough it 21 times. Cleary this is not the right way to do it.
I have a master file with ip addresses one per line. And when I do a dump from the DB to see what nodes are in there, I loop trough those to file to identify indentical ip addresses. The addresse that not match should be excluded from the DB.

The first part of the script is working, but when I wont to exclude the identicals it dint seem to work with awk. Have you someother idé!
Best Reagards Stefan
Hein van den Heuvel
Honored Contributor

Re: Shell script dont work with awk.


>> How can "match" do a match between 1.1.1.1 and 121.192.4.5 if I use a dedicated ip address as regexp?

Because the '.' is a wildcard.
You need to replace for example by [.] for a real dot.

Or do a simple string == as suggested.

Some examples below.

Hein.

$ awk 'BEGIN { print match("121.191.4.5","1.1.1.1")}'
1
$ awk 'BEGIN { print match("121.191.4.5","1[.]1.1.1")}'
0
$ awk 'BEGIN { print match("1.1.1.1","1[.]1.1.1")}'
1
$ awk 'BEGIN { print match("1x1.1.1","1[.]1.1.1")}'
0

Administrator@HP39622780168 ~
$ awk 'BEGIN { print match("1x1.1.1","1.1.1.1")}'
1
Dennis Handly
Acclaimed Contributor

Re: Shell script dont work with awk.

>I have a master file with IP addresses one per line. And when I do a dump from the DB to see what nodes are in there, I loop through those to file to identify indentical IP addresses. The address that not match should be excluded from the DB.

If you are going to use "file" you need to name it every place you mention it, so we don't get confused.

As Hein asked, give us some example files.

>Hein: You need to replace for example by [.] for a real dot.

Or quote them: 1\.1\.1\.1
Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

Ok, I send you the entire script.
And also the inputfile used as first argument (InPut.txt And the second argument file (seedFile.txt

//Stefan
Hein van den Heuvel
Honored Contributor

Re: Shell script dont work with awk.

Dunno what to do with a .rar file. (ok, I do, but I can not right now).
Too specific. zip would be ok, but a plain .TXT file with indication of the inputs, and a example of the desired output MATCHING that input would work best for me, and maybe for others.
Hein.


Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

Ok, InPut.txt is a plain text file with IP-addresses.
10.45.6.8
192.56.7.9
....
....
....
The file to compare with is a output file from a DB, same deal
10.4.5.7
...
...
...
The script reads those two files into arrays.
And a while loop just reads every line from one file and trys to find a match in the other file. If a match is found it directs the result to the file Identical.txt. That are later used to exclude all identical IP-addresses so I get a file with IP-addresses that should not be in the DB.

Dont know how to explain it more :-(
See attached ZIP file!
Best regards Stefan
Peter Nikitka
Honored Contributor

Re: Shell script dont work with awk.

Hi Stefan,

the term we are trying to explain is the difference between 'match' and 'equal'.
The line
awk 'NR==FNR {ip[$1]=$1} NR>FNR {if(ip[$1]) print "double ip",$1}' seedFile.txt InPut.txt

will NOT do a match, but output equal col1 values.

mfG Peter
The Universe is a pretty big place, it's bigger than anything anyone has ever dreamed of before. So if it's just us, seems like an awful waste of space, right? Jodie Foster in "Contact"
Hein van den Heuvel
Honored Contributor
Solution

Re: Shell script dont work with awk.



The shell script you finally provided is different from what you originally posted.
The awk line has a single '>', and therefor only the output from the final execution will remain.

The match function, will return 0 if there is no match right. And non-0 if it matched ANYWHERE on the line.
You knew that right?

Here is an other example of a fine match which may surprise you:

$ awk 'BEGIN { print match("23.91.181.12","1.1.1.1")}'
5

Anyway, why bother with all this stuff if there is a standard tool ready to do all this??
As I indicated before, just use grep -f

grep -f seedFile.txt InPut.txt > IdenticalIp.txt

grep -v -f seedFile.txt InPut.txt > toBeExcluded.txt

Of course you still have to worry about the . being a wildcard, also for grep.
And you possibly need to worry for the match not being anchored. For example, if someone added a comment :
w.x.y.z # Use to 1.2.3.4

Depending on the grep available you may want use:
-F, --fixed-strings PATTERN is a set of newline-separated strings
-x, --line-regexp force PATTERN to match only whole lines
-w, --word-regexp force PATTERN to match only whole words

Enjoy!
Hein.

Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

Thank you Hein!
Will go for that soultion.
And no! I did not know that awk match worked like that. Thank you for clarifying that to me :-)

Ulf Furn
Regular Advisor

Re: Shell script dont work with awk.

See above!