Languages and Scripting
cancel
Showing results for 
Search instead for 
Did you mean: 

analyzing ipfilter logs, need shell script advice

 
SOLVED
Go to solution
Brian Bartley
Frequent Advisor

analyzing ipfilter logs, need shell script advice

All,
I'm trying to come up with a way to import ipfilter logs into Microsfot Access. One problem is the logs sometimes have an extra column. For example the first two lines have nine columns, and the third line has eight columns:

apr 20 joe 0.0.0.0 2x port 15000 len 20
apr 21 bob 0.0.0.1 3x port 15000 len 25
apr 21 dave 0.0.0.3 port 15000 len 28

The first two lines have a column "2x" and "3x" that don't show up in the third line. This makes it hard to import the file into an Access table. I don't need that column with the *x*, I'd like to remove it from every line that contains the extra column while keeping the rest of the line. A simple "grep -v x filename > output.txt will produce a file with all the lines not containing an "x", but I want those lines, I just want to remove the expression that contains the "x".

What utility can do this? Can sed, awk, or cut go through each line and remove just the expression with the "x" character in it? I need to remove the entire expression such as "2x", "13x", and so forth, while retaining the rest of the information in the line. The columns are separated by spaces. Maybe there's a way to tell cut "check the number of columns, if there are nine then remove the fifth one, otherwise do nothing"?

Also if anyone knows of a tool that can take ipfilter logs and analyze them and create a report please let me know. Thanks,

Brian
Brian Bartley
Campus Card Services
Indiana University
10 REPLIES 10
Steven Schweda
Honored Contributor
Solution

Re: analyzing ipfilter logs, need shell script advice

If it can't be done with "sed", then it's not
worth doing, I always say.

$ echo 'apr 21 dave 0.0.0.3 port 15000 len 28' | sed -e 's/ [0-9]*x / /'
apr 21 dave 0.0.0.3 port 15000 len 28

$ echo 'apr 20 joe 0.0.0.0 2x port 15000 len 20' | sed -e 's/ [0-9]*x / /'
apr 20 joe 0.0.0.0 port 15000 len 20

You could also pretend that you're writing
something like a real computer program, and
use "read" to (try to) suck in nine tokens:

read a b c d e f g h i

use "if" to see if the last one is empty:

if [ -z "$i" ] ; then

and then write out the stuff you'd like,
accordingly.

As usual, there may be more than one way to
solve a problem like this.
James R. Ferguson
Acclaimed Contributor

Re: analyzing ipfilter logs, need shell script advice

Hi Brian:

If appears that it's the fifth column you don't want when there are more then eight columns. Hence:

# awk '{if (NF>8) {$5=""};print}' file

...will snip out the extra data.

Regards!

...JRF...
Steven Schweda
Honored Contributor

Re: analyzing ipfilter logs, need shell script advice

> ...will snip out the extra data.

But it leaves in that unsightly extra space.

> [...] sed -e 's/ [0-9]*x / /'

Note that a more complex "sed" command could
effectively determine the token count, and
act accordingly:

$ echo 'aa bb' | \
sed -e 's/^\([^ ][^ ]*\) *[^ ]* *\([^ ][^ ]*\)$/\1 \2/'
aa bb

$ echo 'aa bb cc' | \
sed -e 's/^\([^ ][^ ]*\) *[^ ]* *\([^ ][^ ]*\)$/\1 \2/'
aa cc

Some shortening is possible if your "sed"
accommodates fancy regular expressions, but
I'll admit that it can get ugly.
Brian Bartley
Frequent Advisor

Re: analyzing ipfilter logs, need shell script advice

Steven,
Will this - [0-9]*x - account for any expression that ends in "x" with any number of integers preceeding it? Sometimes it's 2x, or 121x, or even 1232x. Maybe I should leave out the [0-9] and let it operate on any expression that ends in "x"?

Also, when I'm ready to use a file as input could you please give the syntax for the read command, would it be

while read do

done < input file

Thanks,

Brian
Brian Bartley
Campus Card Services
Indiana University
Steven Schweda
Honored Contributor

Re: analyzing ipfilter logs, need shell script advice

> Will this - [0-9]*x - account for any [...]

man sed
man regex

That was the intention. But, by itself,
it'll attack (almost) any token on the line.
And, as shown, you can play with this stuff
interactively, so if you have doubts, then
run the experiment.

> Maybe I should [...]

_You_ get to decide what you really want to
do. There aren't any month name
abbreviations which end in "x", but I don't
know all your users' names. Many things are
possible. Looking for "x" seemed
safer than "x".

> [...] syntax for the read command [...]

What was wrong with:
read a b c d e f g h i
?

man

Look for "read".

There are oodles of shell scripting primers
out there on this new-fangled Inter-Web
thing.
James R. Ferguson
Acclaimed Contributor

Re: analyzing ipfilter logs, need shell script advice

Hi (again) Brian:

> Steven: But it leaves in that unsightly extra space.

But who cares given that the ultimate goal is to import the filtered log into Microsoft software?

If that truly bothers you, you could do:

# awk '{if (NF>8) {$5=""};print}' file|tr -s " "

Regards!

...JRF...
Raj D.
Honored Contributor

Re: analyzing ipfilter logs, need shell script advice

# awk '{if ($5~"x$") {$5=""} print $0}' file
" If u think u can , If u think u cannot , - You are always Right . "
Steven Schweda
Honored Contributor

Re: analyzing ipfilter logs, need shell script advice

> But who cares [...]

I said "unsightly", not "fatal".
Brian Bartley
Frequent Advisor

Re: analyzing ipfilter logs, need shell script advice

Thanks for all replies, once again this forum has helped me considerably.

Brian
Brian Bartley
Campus Card Services
Indiana University