topic Re: grep problem - matching 2 different patterns in Operating System - HP-UX

grep problem - matching 2 different patterns

Danny Fang — Tue, 27 Sep 2005 02:14:12 GMT

Hi,
I'm attempting to "grep" 2 different patterns from a file. The patterns are:
1)flowTargetPort
2)flowSourcePort
3)flowNextHop

The file containing the 3 patterns above has the format of:
flowRawRecord
{
index "0"
flowType "CiscoV1"
flowUptime "2662858776"
flowSequence "8"
flowTOS "0"
flowTCPFlag "16"
flowProtocolNumber "1"
flowInputIfIndex "4"
flowOutputIfIndex "1"
flowNextHop "155.226.201.148"
flowSourceIP "192.168.205.10"
flowTargetIP "155.226.201.148"
flowSourceName "192.168.205.10"
flowTargetName "mola.metricauk.adc.com"
flowSourcePort "0"
flowTargetPort "771"
flowPackets "2"
flowOctets "162"
}

I've used technique shown below to obtain the first 2 patterns i.e. flowTargetPort and flowSourcePort:
bash-2.03$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data |grep "flow.*Port \"123\"" |wc -l
241

However, it failed to return any number of records from the file when I attempted to match the 3rd pattern along with the 1st 2 patterns:
bash-2.03$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data |grep "flow.*Port \"123\""|grep "flowNextHop" |wc -l
0

Could anyone show me how I could use "grep" to match 2 or more patterns simulataneously?

Thanks
Danny

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 02:22:05 GMT

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 02:24:51 GMT

Use simply as,

# grep -E 'flowTargetPort|flowSourcePort|flowNextHop' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | wc -l

or

# egrep 'flowTargetPort|flowSourcePort|flowNextHop' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | wc -l

It will return all counts.

P.S: Don't use cat | grep operation. It is very slow. You can directly use as,

grep

or

grep pattern < file

hth.

Re: grep problem - matching 2 different patterns

Arunvijai_4 — Tue, 27 Sep 2005 02:30:22 GMT

You can use egrep to match 2 or more patterns.

-Arun

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 02:30:43 GMT

You can use awk or sed as well as.

# sed -n '/flowNextHop/p;/flowTargetPort/p;/flowSourcePort/p' | wc -l

or

# awk '/flowTargetPort/ /flowSourcePort/ /flowNextHop/ { print; }' | wc -l

hth.

Re: grep problem - matching 2 different patterns

Raj D. — Tue, 27 Sep 2005 04:05:18 GMT

Hi Danny ,

You can use like this :

$ cat EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data | grep -e "flow.*Port \"123\"" -e "flowNextHop" | wc -l

Syntax:
$ cat filename | grep -e "Pattern1" -e "Pattern2" -e "Pattern_n" | wc -l

Enjoy..
Cheers,
Raj.

Re: grep problem - matching 2 different patterns

Arunvijai_4 — Tue, 27 Sep 2005 04:11:18 GMT

As Muthukumar said, both "cat" and "grep" in a single script will cause performance issues. Try using "grep" directly with the above said options.

-Arun

Re: grep problem - matching 2 different patterns

Sandman! — Tue, 27 Sep 2005 05:17:54 GMT

grep'ing for "flow.*Port \"123\"" won't match the "flowNextHop" pattern. You need to use egrep for that:

# egrep -i "flowTargetPort|flowSourcePort|flowNextHop" EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.dat

regards!

Re: grep problem - matching 2 different patterns

Danny Fang — Tue, 27 Sep 2005 06:08:05 GMT

Hi,

I attempted the awk method suggested by Muthukumar and I obtained the error below:
bash-2.03$ awk '/flowTargetPort/flowSourcePort/flowNextHop/ {print;}' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

May I know where did I go wrong?

Thanks
Danny

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 06:15:48 GMT

It is the problem with your syntax:

Use as:

# awk '/flowTargetPort/ /flowSourcePort/ /flowNextHop/ {print;}' /tmp/datafile
flowNextHop "155.226.201.148"
flowSourcePort "0"
flowTargetPort "771"

# Syntax
# awk '/pattern1//pattern2//pattern3>/ { print; }' < inputfile | wc -l

hth.

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 06:17:52 GMT

You can simply use perl also as,

# perl -ne 'print if /flowTargetPort|flowSourcePort|flowNextHop/ ' EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data

where,

/flowTargetPort|flowSourcePort|flowNextHop/

/ / -> checks for pattern(s).

hth.

Re: grep problem - matching 2 different patterns

Danny Fang — Tue, 27 Sep 2005 06:30:42 GMT

Hi,

If I'd want to include the complete/exact pattern match of the patterns below:
flowSourcePort "0"
flowTargetPort "771"

Could anyone, show me how it's done with both sed and awk?

I've tried using the sed and awk methods below but they did not seem to work.

Usage 1:
bash-2.03$ sed '/flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
Unrecognized command: /flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}

Usage 2:
bash-2.03$ awk '/flowTargetPort \"123\"/flowSourcePort \"123\"/flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

Usage 3:
bash-2.03$ awk '/flowTargetPort \"123\"/ /flowSourcePort \"123\"/ /flowNextHop \"0.0.0.0\/" {print}' EventCollector-evtTest.Test_26_09_2005_04_19_12.data
awk: syntax error near line 1
awk: bailing out near line 1

Thanks
Danny

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 06:35:01 GMT

Why you are missing / between patterns in awk??

# No need to negate " with \ also.

Use like,

awk '/flowTargetPort "123"/ /flowSourcePort "123"/ /flowNextHop "0.0.0.0"/ {print}'

# Example:
# echo 'flowNextHop "0.0.0.0"' | awk '/flowTargetPort "123"/ /flowSourcePort "123"/ /flowNextHop "0.0.0.0"/ {print}'
flowNextHop "0.0.0.0"

hth.

Re: grep problem - matching 2 different patterns

Muthukumar_5 — Tue, 27 Sep 2005 06:38:17 GMT

And why you are using sed with awk syntax??

# Try with sed as,
# sed -n '/flowTargetPort "123"/p;/flowSourcePort "123"/p;/flowNextHop "0.0.0.0"/p'

# Example:
# echo 'flowNextHop "0.0.0.0"' | sed -n '/flowTargetPort "123"/p;/flowSourcePort "123"/p;/flowNextHop "0.0.0.0"/p'
flowNextHop "0.0.0.0"

With perl use as,
perl -ne 'print if /flowTargetPort "123"|flowSourcePort "123"|flowNextHop "0.0.0.0"/'

# Example:
# echo 'flowNextHop "0.0.0.0"' | perl -ne 'print if /flowTargetPort "123"|flowSourcePort "123"|flowNextHop "0.0.0.0"/'
flowNextHop "0.0.0.0"
#

PS: Change your file of EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data to some simple name as test.log for testing.

# simply,
cp EventCollector-evtCJLeaw.CJLeaw_26_09_2005_04_19_12.data /tmp/datafile

It is enough for testing ;)

hth.

Re: grep problem - matching 2 different patterns

Hein van den Heuvel — Tue, 27 Sep 2005 10:06:28 GMT

Danny,

Once you spot those targets, what do you want to do with them?
My guess is that IF you see certain condition THEN you want to print something else that is not part of the conditions.

I would approach that by setting flags for each condition spotted and then examine all flag when the 'end' of a group, or the last condition is spotted.
In a perl script this would look like:

--- filter.pl ----
$nh=$_ if /^flowNextHop/;
$sp=$1 if /^flowSourcePort\s+"(.*)"/;
$tp=$1 if /^flowTargetPort\s+"(.*)"/;
print $nh if (/^}$/ && $sp==0 && $tp==771);

Usage: perl -n filter.pl your-data

Next I suspect that you are not done with just the test/filters mentioned.
I would suggest to generalize that.
For example

---- general-filter.pl -----

$val{$1}=$2 if (/^(\w+)\s*"(.*)"/);
if (/^}$/) {
print $val{flowNextHop}."\n" if
($val{flowSourcePort} == 0
&& $val{flowTargetPort}==771);
}

Usage: perl -n filter.pl your-data

That first line takes anything that vaguely looks like:

If it sees that, it assigns an array element called 'word; to become 'some text'.
For each line.

Now when it see a closing curly on its own it si time to evaluate what has been gathered:
test specific value and print whatever is needed!

hth,
Hein.

Re: grep problem - matching 2 different patterns

Sandman! — Tue, 27 Sep 2005 13:50:38 GMT

Try the following awk construct:

# awk '
> /^flowTargetPort/
> /^flowSourcePort/
> /^flowNextHop/
> ' your_input_file

Re: grep problem - matching 2 different patterns

Jeff Schussele — Tue, 27 Sep 2005 14:01:15 GMT

Hi Danny,

Use grep as follows:

grep -e "pattern one" -e "pattern two" source_file

Rgds,
Jeff

Re: grep problem - matching 2 different patterns

Jeff Lightner_1 — Wed, 28 Sep 2005 08:45:25 GMT

If I read this correctly you not only want to get the pattern but also ONLY the pattern (that is to say you don't what the rest of the same line the pattern appears on). If so you combine awk and grep:

grep -E "flowTargetPort|flowSourcePort|flowNextHop" |awk '{print $1}'

Note the above is all one line.

The "grep -E" (a/k/a egrep) says to do a regexp grep. The items in the quote are separated by a pipe sign which in regexp is used for "or" symbol.

You then pipe the grep -E output (pipe now being used in its shell sense as a form of redirection) to the awk command.

awk by default uses white space (tabs and/or spaces) as the field delimiter). Since the pattern you want is the first item on the line it is the first field so the print command is saying to display the first field.