Looking for SIMPLE log search and filter script for alarms.

[Steve Post]

I am looking for a SIMPLE script. I'll define what I am looking for in successively more complex levels.
----------------------
Level 1: Search-and-filter
I have a big text file that gets more lines in it each day.
I want to note any NEW lines with "critical error" in it.
I want to ignore any NEW lines with "normal stuff" in it.
If I still have stuff, I would probably raise an alarm.
----------------------
Level 2: setof-Search expression setof-filter expressions set of files.
Remember I said a big text file? Now I mean a set of text files in a list.

Remember I said "critical error"? Now I mean a set of regular expressions like "critical error" "disk full" "illegal user" "bad stuff goin on" "job died"

Remember I said "normal stuff"? Now I mean a set of regular expression filters like:
"cronjob one.sh successful" "received file" "job completed fine"
----------------------
Level 3: Atleast-one-occurance, or raise a flag.
Remember I said "normal stuff" that I filter? Now I mean to look for at least one instance of the normal stuff since the last time I ran the job. If this is not present, there must be a problem.
So I every time I run the program, I expect to see "job complete fine" at least once, or I assume there is an error, and raise a flag.
----------------------
Level 4: the-I-already-saw-it filter (i.e. a mute button)
Add one more filter. If I already saw the "expression" I am searching for, I don't want to be notified about it again, until I clear the notice.
----------------------
What I am NOT looking for. Anything that comes in an acronym. Anything that requires a gazillion addition pieces of stuff to install. Anything that requires compiling. Anything that is not simple text. Anything that costs money. I figure someone has already gone through this exercise with shell or perl scripting.

[Tim Nelson]

Here is the summary of what I use.

create logcheck script.
first time through it saves the number of lines that the file you are watching has.

Next time through subtracts the previous from current. Uses that number in tail. Pipes through egrep.

egrep uses an exclude file with all your search strings you do not want.

Then sends the difference via email.

There is also a check that if the current line count is less than previous the either you cleared the file or someone is tampering .

I use this to monitor the syslog.log

I can send examples if you would like.

[Steve Post]

Thanks. I was hoping for a discussion on various cody algorthyms to use when doing this.
For example for level1 idea.
cat logfile | grep -Ei "failed for illegal|Access denied" | \
grep -Ev "spam|FTP|Data port 20"

for level2:
I have LOGLIST hardwired to be a set of log files. I have filter.txt and search.txt as text files where there is one string on each line in the files.

for log in $LOGLIST
do
cp -p ${log} ${log}.tmp

# filter out stuff from ${log}.tmp
for filter in `cat ./filter.txt | grep -Ev "^#"`
do
cat ${log}.tmp | grep -vi $filter > ${log}.tmp2
mv ${log}.tmp2 ${log}.tmp
done

# search for stuff in ${log}.tmp
for search in `cat ./search.txt`
do
cat ${log}.tmp | grep -vi $search > ${log}.tmp2
done
mv ${log}.tmp2 ${log}.tmp
ls -l ${log} ${log}.tmp
done

This level2 works reasonably. But it has a bug. If I am searching for "evil user", it will search for "evil", then "user".

[Tim Nelson]

Here is the jist. The IGNORE file is a list of messages or seach strings to ignore. It could be combined with a number of other twists.

### MAIN

trackcnt=`cat $trackfile`

if [[ $syslogcnt = $trackcnt ]]
then
#Nothing to do, no changes
#echo "Nothing to do, no changes"
exit 0
else
if [[ $trackcnt -gt $syslogcnt ]]
then
# Must be new syslog or somebody is messing
SUBJECT="ALERT - $HOSTNAME Track count is greater than syslog file"
trackcnt=$syslogcnt
tail -$trackcnt $syslog|egrep -vf $IGNORE|mailx -s "$SUBJECT" $EMAIL
else
# Execute search
(( diffcnt = syslogcnt - trackcnt ))
SUBJECT="INFO - $HOSTNAME"
tail -$diffcnt $syslog|egrep -vf $IGNORE > $tmpfile
if [[ -s $tmpfile ]]
then
tail -$diffcnt $syslog|egrep -vf $IGNORE|mailx -m -s "$SUBJECT" $EMAIL1
fi
fi
fi

[Joel Girot]

For file monitoring and analysis, I use the free open source tool "SEC - simple event correlator" (author Risto Vaarandi) a very good tool for real-time analyse of various log files with immediate notification. More at http://kodu.neti.ee/~risto/sec/
Joel

[Steve Post]

Joel, I'm NOT looking for a gee-whiz package. I'm looking for the concepts in scripting. Actually I already found at least one package: Logfile-1.0.2. I'm looking into it's contents as a guide.

Tim,
don't you mean
trackcnt=`cat $trackfile | wc -l | awk '// {print $1}'`?
To put it another way, don't you mean that trackcnt should be the number of lines in file trackfile?
Or are you really holding the contents on the entire file in a variable?

fyi egrep has been replaced with grep -E.

and myself....I meant codING, not codY.

And on my sample problem of how to make "evil user" search for "evil user" and not "evil" .... "user"?
I can use
exec 3while read X 0<&3
do
cat logfile | grep -Ev "$X" > logfile.cleaned
done
3<&-

[Tim Nelson]

Steve,

Yes trackcnt is used as a hold for the number of lines in the file.

The idea is to only tail the last lines since the previous check.

The previous line count is held in a file so I can run this via cron every 15 minutes instead of continuously with a sleep.