topic Re: help in removing entries from a file in Operating System - Linux

help in removing entries from a file

lawrenzo_1 — Mon, 03 Sep 2007 05:06:00 GMT

Guys,

I could do with some help:

I have a file with a list of entries that I want to remove from a logfile:

iprange1
iprange2
iprange3
iprange4

what would be the best way to remove these from a logfile with many ipranges in?

Thanks

Chris

Re: help in removing entries from a file

Oviwan — Mon, 03 Sep 2007 05:12:16 GMT

Hey

something like that:

$cat logfile | grep -v ipranges > temp.file
$cat temp.file > logfile

or what syntax have your iprange?

Regards

Re: help in removing entries from a file

whiteknight — Mon, 03 Sep 2007 05:17:31 GMT

Hi Lawrenzo,

See also
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1156808

WK

Re: help in removing entries from a file

lawrenzo_1 — Mon, 03 Sep 2007 06:13:13 GMT

ok yes I know these options however what I am looking for is instead of doing:

grep -v iprange1 |grep -v iprange2 |grep -v iprange3 etc etc

is have a list of each different iprange that I do not what to display from the logfile in a file then use this to not display from the log:

ie

how do I have a file with

iprange1
iprange2
iprange3
iprange4

then not display this information from the logfile that has multiple entries of

iprange1
iprange2
iprange3
iprange4
iprange5
iprange6
iprange7
iprange8

basically the ipranges are exeptions from a report that is emailed daily.

Thanks chaps.

Re: help in removing entries from a file

James R. Ferguson — Mon, 03 Sep 2007 06:59:41 GMT

Hi Chris:

# grep -v -f pattern_file data_file

...where 'pattern_file' contains the strings that you do *not* want to include in your output.

Please consider also evaluating your previous thread on this subject:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1156808

Regards!

...JRF...

Re: help in removing entries from a file

James R. Ferguson — Mon, 03 Sep 2007 08:07:19 GMT

Hi (again) Chris:

Actually for *ranges* you have 'grep's ability to use extended regular expressions ('-E') in your pattern file ('-f)' [see the manpges for 'grep']:

For example:

# cat .pattern_file
10.11.*.2

# cat .data_file
10.11.12.1
10.11.12.2
10.11.12.3
10.11.13.1
10.11.13.2
10.11.13.3
10.11.14.1
10.11.14.2
10.11.14.3

# grep -v -E -f .pattern_file data_file
10.11.13.1
10.11.13.3
10.11.14.1
10.11.14.3

Regards!

...JRF...

Re: help in removing entries from a file

James R. Ferguson — Mon, 03 Sep 2007 08:48:01 GMT

Hi Chris:

DISREGARD MY LAST POST! ENOCOFFEE! We need to escape the 'dot' if we want it to mean itself and not any character in the context I presented.

Actually for *ranges* you have 'grep's ability to use extended regular expressions ('-E') in your pattern file ('-f)' [see the manpges for 'grep']:

For example:

# cat .pattern_file
10\.11\.[0-9]*\.2

# cat .data_file
10.11.12.1
10.11.12.2
10.11.12.3
10.11.13.1
10.11.13.2
10.11.13.3
10.11.14.1
10.11.14.2
10.11.14.3

# grep -v -E -f ./pattern_file ./data_file
10.11.12.1
10.11.12.3
10.11.13.1
10.11.13.3
10.11.14.1
10.11.14.3

# grep -E -f ./pattern_file ./data_file
10.11.12.2
10.11.13.2
10.11.14.2

Regards!

...JRF...

Re: help in removing entries from a file

Peter Nikitka — Mon, 03 Sep 2007 11:46:49 GMT

Hi,

just as JRF remarked: make yourself clear, if you want to consider 'iprange1' as string or as pattern.
Note, that using it as pattern creates the problem, that you have to watch for wildcard characters: something like
1.2.3.4
will match
152.3.4
as well, because '.' is 'any char' when using it in regular expressions! Best use
1[.]2[.]3[.]4
for such a case.
Important as well is the fact, if your pattern are anchored at begin or end of a line. Taking this into account can speed up your script dramatically!

Second: NOT wanting a pattern match, use
fgrep (or grep -F) - this does pure string compares.

mfG Peter

Re: help in removing entries from a file

lawrenzo_1 — Mon, 03 Sep 2007 11:59:23 GMT

Thanks all,

I am not too familiar with fgrep or egrep however do understand these are powerful tools and will check and test out.

Chris.

Re: help in removing entries from a file

Dennis Handly — Mon, 03 Sep 2007 16:47:19 GMT

>JRF: extended regular expressions

No need to use the -E hammer, your regular expressions (*) aren't extended.

>Peter: Best use 1[.]2[.]3[.]4

Usually they are quoted with a "\".

>if your pattern are anchored at begin or end of a line. Taking this into account can speed up your script dramatically!

Right. You can do that with grep -x but the logfile could only have that number and no other data on each line.

I'm surprised Peter didn't mention the gotcha with fgrep, 1.2.3.4 will match 21.2.3.4

So you need to use grep -w, or use anchored strings in the file. Given't JRF's file:
10\.11\.[0-9]*\.2

You could use vi to add the anchors:
:%s/^/^/
:%s/$/$/