grep problem - matching 2 different patterns

[Danny Fang]

Hi,
I'm attempting to "grep" 2 different patterns from a file. The patterns are:
1)flowTargetPort
2)flowSourcePort
3)flowNextHop

The file containing the 3 patterns above has the format of:
flowRawRecord
{
index "0"
flowType "CiscoV1"
flowUptime "2662858776"
flowSequence "8"
flowTOS "0"
flowTCPFlag "16"
flowProtocolNumber "1"
flowInputIfIndex "4"
flowOutputIfIndex "1"
flowNextHop "155.226.201.148"
flowSourceIP "192.168.205.10"
flowTargetIP "155.226.201.148"
flowSourceName "192.168.205.10"
flowTargetName "mola.metricauk.adc.com"
flowSourcePort "0"
flowTargetPort "771"
flowPackets "2"
flowOctets "162"
}

I've used technique shown below to obtain the first 2 patterns i.e. flowTargetPort and flowSourcePort:
bash-2.03$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data |grep "flow.*Port \"123\"" |wc -l
241

However, it failed to return any number of records from the file when I attempted to match the 3rd pattern along with the 1st 2 patterns:
bash-2.03$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data |grep "flow.*Port \"123\""|grep "flowNextHop" |wc -l
0

Could anyone show me how I could use "grep" to match 2 or more patterns simulataneously?

Thanks
Danny

[Muthukumar_5 · Accepted solution]

You can use grep -E "pattern1|pattern2|..|patternn" or simply egrep "pattern1|pattern2|..|patternn"

# Example's
# echo "hi\nbye" | grep -E 'hi|bye'
hi
bye
# echo "hi\nbye" | egrep 'hi|bye'
hi
bye
#

hth.

[Muthukumar_5]

Use simply as,

# grep -E 'flowTargetPort|flowSourcePort|flowNextHop' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | wc -l

or

# egrep 'flowTargetPort|flowSourcePort|flowNextHop' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | wc -l

It will return all counts.

P.S: Don't use cat | grep operation. It is very slow. You can directly use as,

grep

or

grep pattern < file

hth.

[Arunvijai_4]

You can use egrep to match 2 or more patterns.

-Arun

[Muthukumar_5]

You can use awk or sed as well as.

# sed -n '/flowNextHop/p;/flowTargetPort/p;/flowSourcePort/p' | wc -l

or

# awk '/flowTargetPort/ /flowSourcePort/ /flowNextHop/ { print; }' | wc -l

hth.

[Raj D.]

Hi Danny ,

You can use like this :

$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | grep -e "flow.*Port \"123\"" -e "flowNextHop" | wc -l


Syntax:
$ cat filename | grep -e "Pattern1" -e "Pattern2" -e "Pattern_n" | wc -l

Enjoy..
Cheers,
Raj.

[Arunvijai_4]

As Muthukumar said, both "cat" and "grep" in a single script will cause performance issues. Try using "grep" directly with the above said options.

-Arun

[Sandman!]

grep'ing for "flow.*Port \"123\"" won't match the "flowNextHop" pattern. You need to use egrep for that:

# egrep -i "flowTargetPort|flowSourcePort|flowNextHop" EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.dat

regards!

[Danny Fang]

Hi,

I attempted the awk method suggested by Muthukumar and I obtained the error below:
bash-2.03$ awk '/flowTargetPort/flowSourcePort/flowNextHop/ {print;}' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

May I know where did I go wrong?

Thanks
Danny

[Muthukumar_5]

It is the problem with your syntax:

Use as:

# awk '/flowTargetPort/ /flowSourcePort/ /flowNextHop/ {print;}' /tmp/datafile
flowNextHop "155.226.201.148"
flowSourcePort "0"
flowTargetPort "771"

# Syntax
# awk '/pattern1//pattern2//pattern3>/ { print; }' < inputfile | wc -l

hth.

[Muthukumar_5]

You can simply use perl also as,

# perl -ne 'print if /flowTargetPort|flowSourcePort|flowNextHop/ ' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data

where,

/flowTargetPort|flowSourcePort|flowNextHop/

/ / -> checks for pattern(s).


hth.

[Danny Fang]

Hi,

If I'd want to include the complete/exact pattern match of the patterns below:
flowSourcePort "0"
flowTargetPort "771"

Could anyone, show me how it's done with both sed and awk?

I've tried using the sed and awk methods below but they did not seem to work.

Usage 1:
bash-2.03$ sed '/flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
Unrecognized command: /flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}

Usage 2:
bash-2.03$ awk '/flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

Usage 3:
bash-2.03$ awk '/flowTargetPort \"123\"/ /flowSourcePort \"123\"/ /flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

Thanks
Danny

[Muthukumar_5]

Why you are missing / between patterns in awk??

# No need to negate " with \ also.

Use like,

awk '/flowTargetPort "123"/ /flowSourcePort "123"/ /flowNextHop "0.0.0.0"/ {print}'

# Example:
# echo 'flowNextHop "0.0.0.0"' | awk '/flowTargetPort "123"/ /flowSourcePort "123"/ /flowNextHop "0.0.0.0"/ {print}'
flowNextHop "0.0.0.0"

hth.

[Muthukumar_5]

And why you are using sed with awk syntax??

# Try with sed as,
# sed -n '/flowTargetPort "123"/p;/flowSourcePort "123"/p;/flowNextHop "0.0.0.0"/p'

# Example:
# echo 'flowNextHop "0.0.0.0"' | sed -n '/flowTargetPort "123"/p;/flowSourcePort "123"/p;/flowNextHop "0.0.0.0"/p'
flowNextHop "0.0.0.0"

With perl use as,
perl -ne 'print if /flowTargetPort "123"|flowSourcePort "123"|flowNextHop "0.0.0.0"/'

# Example:
# echo 'flowNextHop "0.0.0.0"' | perl -ne 'print if /flowTargetPort "123"|flowSourcePort "123"|flowNextHop "0.0.0.0"/'
flowNextHop "0.0.0.0"
#

PS: Change your file of EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data to some simple name as test.log for testing.

# simply,
cp EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data /tmp/datafile

It is enough for testing ;)

hth.

[Hein van den Heuvel]

Danny,

Once you spot those targets, what do you want to do with them?
My guess is that IF you see certain condition THEN you want to print something else that is not part of the conditions.

I would approach that by setting flags for each condition spotted and then examine all flag when the 'end' of a group, or the last condition is spotted.
In a perl script this would look like:

--- filter.pl ----
$nh=$_ if /^flowNextHop/;
$sp=$1 if /^flowSourcePort\s+"(.*)"/;
$tp=$1 if /^flowTargetPort\s+"(.*)"/;
print $nh if (/^}$/ && $sp==0 && $tp==771);


Usage: perl -n filter.pl your-data


Next I suspect that you are not done with just the test/filters mentioned.
I would suggest to generalize that.
For example

---- general-filter.pl -----

$val{$1}=$2 if (/^(\w+)\s*"(.*)"/);
if (/^}$/) {
print $val{flowNextHop}."\n" if
($val{flowSourcePort} == 0
&& $val{flowTargetPort}==771);
}

Usage: perl -n filter.pl your-data

That first line takes anything that vaguely looks like:







If it sees that, it assigns an array element called 'word; to become 'some text'.
For each line.

Now when it see a closing curly on its own it si time to evaluate what has been gathered:
test specific value and print whatever is needed!

hth,
Hein.

[Sandman!]

Try the following awk construct:

# awk '
> /^flowTargetPort/
> /^flowSourcePort/
> /^flowNextHop/
> ' your_input_file

[Jeff Schussele]

Hi Danny,

Use grep as follows:

grep -e "pattern one" -e "pattern two" source_file

Rgds,
Jeff

[Jeff Lightner_1]

If I read this correctly you not only want to get the pattern but also ONLY the pattern (that is to say you don't what the rest of the same line the pattern appears on). If so you combine awk and grep:

grep -E "flowTargetPort|flowSourcePort|flowNextHop" |awk '{print $1}'

Note the above is all one line.

The "grep -E" (a/k/a egrep) says to do a regexp grep. The items in the quote are separated by a pipe sign which in regexp is used for "or" symbol.

You then pipe the grep -E output (pipe now being used in its shell sense as a form of redirection) to the awk command.

awk by default uses white space (tabs and/or spaces) as the field delimiter). Since the pattern you want is the first item on the line it is the first field so the print command is saying to display the first field.