topic Re: analyzing ipfilter logs, need shell script advice in Operating System - HP-UX

analyzing ipfilter logs, need shell script advice

Brian Bartley — Tue, 20 Apr 2010 13:24:20 GMT

All,
I'm trying to come up with a way to import ipfilter logs into Microsfot Access. One problem is the logs sometimes have an extra column. For example the first two lines have nine columns, and the third line has eight columns:

apr 20 joe 0.0.0.0 2x port 15000 len 20
apr 21 bob 0.0.0.1 3x port 15000 len 25
apr 21 dave 0.0.0.3 port 15000 len 28

The first two lines have a column "2x" and "3x" that don't show up in the third line. This makes it hard to import the file into an Access table. I don't need that column with the *x*, I'd like to remove it from every line that contains the extra column while keeping the rest of the line. A simple "grep -v x filename > output.txt will produce a file with all the lines not containing an "x", but I want those lines, I just want to remove the expression that contains the "x".

What utility can do this? Can sed, awk, or cut go through each line and remove just the expression with the "x" character in it? I need to remove the entire expression such as "2x", "13x", and so forth, while retaining the rest of the information in the line. The columns are separated by spaces. Maybe there's a way to tell cut "check the number of columns, if there are nine then remove the fifth one, otherwise do nothing"?

Also if anyone knows of a tool that can take ipfilter logs and analyze them and create a report please let me know. Thanks,

Brian

Re: analyzing ipfilter logs, need shell script advice

Steven Schweda — Tue, 20 Apr 2010 13:49:47 GMT

If it can't be done with "sed", then it's not
worth doing, I always say.

$ echo 'apr 21 dave 0.0.0.3 port 15000 len 28' | sed -e 's/ [0-9]*x / /'
apr 21 dave 0.0.0.3 port 15000 len 28

$ echo 'apr 20 joe 0.0.0.0 2x port 15000 len 20' | sed -e 's/ [0-9]*x / /'
apr 20 joe 0.0.0.0 port 15000 len 20

You could also pretend that you're writing
something like a real computer program, and
use "read" to (try to) suck in nine tokens:

read a b c d e f g h i

use "if" to see if the last one is empty:

if [ -z "$i" ] ; then

and then write out the stuff you'd like,
accordingly.

As usual, there may be more than one way to
solve a problem like this.

Re: analyzing ipfilter logs, need shell script advice

James R. Ferguson — Tue, 20 Apr 2010 13:53:54 GMT

Hi Brian:

If appears that it's the fifth column you don't want when there are more then eight columns. Hence:

# awk '{if (NF>8) {$5=""};print}' file

...will snip out the extra data.

Regards!

...JRF...

Re: analyzing ipfilter logs, need shell script advice

Steven Schweda — Tue, 20 Apr 2010 14:25:15 GMT

> ...will snip out the extra data.

But it leaves in that unsightly extra space.

> [...] sed -e 's/ [0-9]*x / /'

Note that a more complex "sed" command could
effectively determine the token count, and
act accordingly:

$ echo 'aa bb' | \
sed -e 's/^$[^ ][^ ]*$ *[^ ]* *$[^ ][^ ]*$$/\1 \2/'
aa bb

$ echo 'aa bb cc' | \
sed -e 's/^$[^ ][^ ]*$ *[^ ]* *$[^ ][^ ]*$$/\1 \2/'
aa cc

Some shortening is possible if your "sed"
accommodates fancy regular expressions, but
I'll admit that it can get ugly.

Re: analyzing ipfilter logs, need shell script advice

Brian Bartley — Tue, 20 Apr 2010 14:38:25 GMT

Steven,
Will this - [0-9]*x - account for any expression that ends in "x" with any number of integers preceeding it? Sometimes it's 2x, or 121x, or even 1232x. Maybe I should leave out the [0-9] and let it operate on any expression that ends in "x"?

Also, when I'm ready to use a file as input could you please give the syntax for the read command, would it be

while read do

done < input file

Thanks,

Brian

Re: analyzing ipfilter logs, need shell script advice

Steven Schweda — Tue, 20 Apr 2010 15:08:40 GMT

> Will this - [0-9]*x - account for any [...]

man sed
man regex

That was the intention. But, by itself,
it'll attack (almost) any token on the line.
And, as shown, you can play with this stuff
interactively, so if you have doubts, then
run the experiment.

> Maybe I should [...]

_You_ get to decide what you really want to
do. There aren't any month name
abbreviations which end in "x", but I don't
know all your users' names. Many things are
possible. Looking for "x" seemed
safer than "x".

> [...] syntax for the read command [...]

What was wrong with:
read a b c d e f g h i
?

man

Look for "read".

There are oodles of shell scripting primers
out there on this new-fangled Inter-Web
thing.

Re: analyzing ipfilter logs, need shell script advice

James R. Ferguson — Tue, 20 Apr 2010 15:10:53 GMT

Hi (again) Brian:

> Steven: But it leaves in that unsightly extra space.

But who cares given that the ultimate goal is to import the filtered log into Microsoft software?

If that truly bothers you, you could do:

# awk '{if (NF>8) {$5=""};print}' file|tr -s " "

Regards!

...JRF...

Re: analyzing ipfilter logs, need shell script advice

Raj D. — Tue, 20 Apr 2010 15:17:28 GMT

# awk '{if ($5~"x$") {$5=""} print $0}' file

Re: analyzing ipfilter logs, need shell script advice

Steven Schweda — Tue, 20 Apr 2010 15:49:35 GMT

> But who cares [...]

I said "unsightly", not "fatal".

Re: analyzing ipfilter logs, need shell script advice

Brian Bartley — Tue, 20 Apr 2010 16:15:23 GMT

Thanks for all replies, once again this forum has helped me considerably.

Brian

Re: analyzing ipfilter logs, need shell script advice

Dennis Handly — Tue, 20 Apr 2010 23:59:08 GMT

If you want to remove that column, you can also re-print out each line
awk '
NF == 8 { print $0; next }
{ print $1, $2, $3, $4, $6, $7, $8, $9 }' file