Shell script dont work with awk.

[Ulf Furn]

Hi everybody. I hope that some one can help me with this,
I trying to identify identical ip addresses by comparing to file, this is not a problem.

The problem is that i cant seem to read the values back to a file.

This is the code I use!

IDENTICAL=(`cat IdenticalIp.txt`)
IDENTRIES=${#IDENTICAL[@]}

while [ $index3 -lt $IDENTRIES ]
do
awk 'match($0, "${IDENTICAL[index3]}") == 0 {print$0}' $OVIP >> toBeExcluded.txt
echo "${IDENTICAL[index3]}"
index3=$((index3+1))
done

The awk command works when I run it from the command line and use a ip address insted of my array. it seems that awk dont handle the array variable correct! HELP!
Best regards //Stefan

[Hein van den Heuvel]

The shell variable ${index3} never makes it into your awk construct.
The value for index3 within awk will be undefined / 0.

You can fix that with more a more tricky argument line, doing the substitution before getting into awk, or you can just pass the variable explicitly into awk:

$ test="aap"
$ awk -vtest=$test 'BEGIN {print test}'
aap

-vindex3=$index3

? and how is that array getting into awk?


But why not just in awk and do everything that needs to be done right there?
Just suck that indentical file into an awk array in its 'BEGIN' section.

Something like:
$ cat > tmp.txt
aap
noot
mies
$ awk 'BEGIN { while (getline lines[i++] < "tmp.txt"); print lines[2],lines[1],lines[0]}'

mies noot aap


For you that would like like:

$ awk 'BEGIN { while (getline IDENTICAL[i++] < "IdenticalIp.tx");} { .. main code body .. }' $OVIP >> toBeExcluded.txt

Please provide a few useful example lines of the two input file and a corresponding output file if more help is needed.

Good luck!
Hein.

[James R. Ferguson]

Hi Stefan:

Pass the array element to 'awk':

# awk -v element=${IDENTICAL[index3]} 'match($0, element) == 0 {print $0}'

Regards!

...JRF...

[Dennis Handly]

>IDENTICAL=(`cat IdenticalIp.txt`)

No need for () nor evil cats:
IDENTICAL=$(< IdenticalIp.txt)

Unless you have made IDENTICAL an array, it isn't:
set -A IDENTICAL $(< IdenticalIp.txt)

>index3=$((index3+1))

This can be done as:
(( index3 += 1 ))

Also instead of reading everything into an array, you can use a while to read out of IdenticalIp.txt:
while read IDENTICAL; do
awk -v IDENTICAL=$IDENTICAL 'match($0, IDENTICAL) == 0 {print$0}' $OVIP >> toBeExcluded.txt
echo "${IDENTICAL}"
done < IdenticalIp.txt

Also there is no need to use match, you should be able to use !~ for matching.

[Ulf Furn]

Hi and thanks for your responses. But it dont work any way.The awk command still dont compare and exclude the identical input.

For input in the loop, its pass 21 addresses.
So when I pass this to awk it outputs it 21 times. Summery 441 lines.
If I only let the loop write to the output file ones. it exclude one address and leaves the rest.

IDENTICAL=(`cat IdenticalIp.txt`) #Containes identical ipaddresses from earlier output.
IDENTRIES=${#IDENTICAL[@]} #Array containes 21 elements

echo "$IDENTRIES"
while [ $index3 -lt $IDENTRIES ]
do
awk -v element=${IDENTICAL[index3]} 'match($0, element) == 0 {print$0}' $OVIP > toBeExcluded.txt
#echo "${IDENTICAL[index3]}"
index3=$((index3+1))
done

# The file $OVIP is the file to compare the addresses. toBeExcluded.txt should only after processeing, contain those addresses that are to be excluded. The identical adressed should not be in the output file.

[Hein van den Heuvel]

Do you really want the lines from IdenticalIp.txt to be treated as regular expression, such as 'match' will?

It will make 1.1.1.1 match 121.191.3.4
Is that desirable or acceptable?

If it is, why not just use:

# grep -v -f IdenticalIp.txt $OVIP > toBeExcluded.txt

If it is not acceptable, then you may need to use something like

substr($0,1,length(element) == element { ...}
of course you could do all that in shell code.

And if you do not check the filter file as you read it, then it should be very certain that it is 'clean'. No trailing spaces, or empty lines to force unexpected matches.

So far this remains an n x m algoritme as coded. (21*21=441).
Is that acceptable at production time?
Is either of the files expected to be much larger than the other?

If you read the filter into an AWK, or PERL associative array as per my first reply, and then use the IP from the second file as a key to attempt an array lookup, then it becomes and n + m algorithm (21 + 21=42).
It will only start up perl/awk once.
Much more efficient! ... if that matters at all

Your example is cute, but too smart for me.
I think it would help get better replies if you provide a few simple test lines from the input files and desired output. Just paste them here, and if spaces are important, also paste them into a .TXT file to be attached.

Hope this helps,
Hein.

[Ulf Furn]

Well, I thought so!
The file that are read in to the array and then walk trough the loop, contains only ip addresses. But when I looking att this more and more, am not sure.

How can "match" do a match between 1.1.1.1 and 121.192.4.5 if I use a dedicated ip address as regexp? I am missing something here. The loop dont just loop trough the 21 entries in the array, it loops trough it 21 times. Cleary this is not the right way to do it.
I have a master file with ip addresses one per line. And when I do a dump from the DB to see what nodes are in there, I loop trough those to file to identify indentical ip addresses. The addresse that not match should be excluded from the DB.

The first part of the script is working, but when I wont to exclude the identicals it dint seem to work with awk. Have you someother idé!
Best Reagards Stefan

[Hein van den Heuvel]

>> How can "match" do a match between 1.1.1.1 and 121.192.4.5 if I use a dedicated ip address as regexp?

Because the '.' is a wildcard.
You need to replace for example by [.] for a real dot.

Or do a simple string == as suggested.

Some examples below.

Hein.

$ awk 'BEGIN { print match("121.191.4.5","1.1.1.1")}'
1
$ awk 'BEGIN { print match("121.191.4.5","1[.]1.1.1")}'
0
$ awk 'BEGIN { print match("1.1.1.1","1[.]1.1.1")}'
1
$ awk 'BEGIN { print match("1x1.1.1","1[.]1.1.1")}'
0

Administrator@HP39622780168 ~
$ awk 'BEGIN { print match("1x1.1.1","1.1.1.1")}'
1

[Dennis Handly]

>I have a master file with IP addresses one per line. And when I do a dump from the DB to see what nodes are in there, I loop through those to file to identify indentical IP addresses. The address that not match should be excluded from the DB.

If you are going to use "file" you need to name it every place you mention it, so we don't get confused.

As Hein asked, give us some example files.

>Hein: You need to replace for example by [.] for a real dot.

Or quote them: 1\.1\.1\.1

[Ulf Furn]

Ok, I send you the entire script.
And also the inputfile used as first argument (InPut.txt And the second argument file (seedFile.txt

//Stefan