- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- Searching Errors in a log file
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2006 05:13 PM
12-09-2006 05:13 PM
I am looking out for a shell script to accomplish the below :-
We want to capture each occurance of the word Error ( case insensitive ) in a log file and 5 lines below each of the lines and store the output in a file.
Can anyone post the script ?
Thanks,
Shiv
Solved! Go to Solution.
- Tags:
- grep
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2006 10:08 PM
12-09-2006 10:08 PM
SolutionSome open issues:
- you say 'word': is it really a separate word (white space around)? 'Error:' for example would NOT meet this criterium.
- you say case insensitive: may the be ANY combinations of Error, ErrOR or is there a restriction to Error, error and ERROR?
I offer an awk solution (untested).
awk '/Error/ || /error/ || /ERROR/ {count=6}
count {print;count--}' infile >outfile
mfG Peter
- Tags:
- awk
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-09-2006 10:15 PM
12-09-2006 10:15 PM
Re: Searching Errors in a log file
try the attached script with these parameters: $1="Error" , $2="5", $3="path_to_file"
regards,
John K.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2006 01:02 AM
12-10-2006 01:02 AM
Re: Searching Errors in a log file
Perl regular expression have something a 'break' match between words: \b
It helps to make sure that a script like requested does trigger on "Error:" but not on "terrorist".
Example usage:
perl -ne "$count=5 if /\berror\b/i; print if $count-- > 0" your-log-file
hth,
Hein.
- Tags:
- Perl
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2006 08:25 AM
12-10-2006 08:25 AM
Re: Searching Errors in a log file
grep -i -e fail -e warn -e alert -e critical -e abort /some-dir/some-log
The above grep just shows the line where one or more of the search strings match. Use it to see if there are any other messages that might be important but do not have Error in the entry.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2006 08:59 AM
12-10-2006 08:59 AM
Re: Searching Errors in a log file
Bill makes a good point, and we can easily use Hein's Perl snippet thusly:
# perl -ne '$count=5 if /\b(error|warn|critical|alert|abort|fail)\b/i;print if $count-- > 0' logfile
Thus if the file (or files) you specify as arguments contain any of the words "error", "warn", "critical", "alert", "abort" or "fail", the line on which any one occurs will be printed and four more lines afterwards.
As written, you can process multiple files at one simply by passing their names as arguments. If you would like to do this, *and* keep track of the filename for which matches occur, modify the script thusly:
# perl -ne '$count=5 if /\b(error|warn|critical|alert|abort|fail)\b/i;print "$ARGV:$_" if $count-- > 0' log1 log2 log3
...Now output can look like:
log1:critical alert: line-3
log1:line-4
log1:line-5
log1:line-6
log1:line-7
log2:ERROR at line-6
log2:line-7
log2:line-8
log2:line-9
log2:line-10
Regards!
..JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2006 02:46 AM
12-11-2006 02:46 AM
Re: Searching Errors in a log file
I put a space before each word.
so it would get things like " Error:"
awk -F: '
(tolower($0) !~ / error| warn| alert| critical| abort| invalid/) {next}
{
print "\n" $0
for (i=1;i < 6;i++)
{getline
print $0
}
} ' /var/adm/syslog/syslog.log
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2006 03:01 AM
12-11-2006 03:01 AM
Re: Searching Errors in a log file
Yes, placing a space (blank) character before the words in the 'awk' solution helps, but it dones *not* solve the number of matches that Perl does.
For starters, what if the space were a tab character? The use of '\b' in Perl regular expressions covers word boundries delimited by spaces, tabs and most punctuation characters like ":" and "-" that might appear like "alert-bad thing happened".
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2006 03:29 AM
12-11-2006 03:29 AM
Re: Searching Errors in a log file
This may not be what you require, but I have used it alot. It will watch your log files and email you whenever it finds any security violations or errors. YOu can tailor it to watch for certain words - like "error"
http://sourceforge.net/projects/logcheck/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2006 04:26 AM
12-11-2006 04:26 AM
Re: Searching Errors in a log file
Jim,
Like all programming problems there are many tools with many solutions. My intent was to show how you can use grep like expressions inside of awk The tolower/toupper function inside the if statement is novel approach, along with the regular expression syntax. Of course, I made the fatal error of assuming that the only type of white space is a "space". Which in the syslogs I looked at was true.
a partial solution:
awk -F: '
(tolower($0) ~ /[ \t]error|[ \t]warn|[ \t]alert|[ \t]critical|[ \t]abort|[ \t]invalid/ ) {
print "\n" $0
for (i=1;i < 6;i++)
{
getline
print $0
}
} ' /var/adm/syslog/syslog.log
To bad I could not get the following to work:
(tolower($0) ~ /[:space:]error|[:space:]warn|[:space:]alert|[:space:]critical|[:space:]abort|[:space:]invalid/)
Rory
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-11-2006 04:46 AM
12-11-2006 04:46 AM
Re: Searching Errors in a log file
I did not mean any offense; rather only that Perl has some of the best regular expression support available. Your use of '[:space:]' enables whitespace (a space or a tab) detection and is good defensive technique. This works:
# awk '{if (tolower($0) ~/[[:space:]]error|[[:space:]]warn/) {print $0}}' filename
Note the double square brackets. This is necessary to create a character class consisting of '[:space:]'.
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2006 04:00 AM
12-12-2006 04:00 AM
Re: Searching Errors in a log file
No offense was taken and I hope my reply did not suggest that I was offended. The discussion about problems and solutions are always good.
Rory
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2006 11:20 AM
12-12-2006 11:20 AM
Re: Searching Errors in a log file
Is this the right syntax to use your script ?
$scriptname.sh "Error" "5" "logfilename"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2006 05:25 AM
12-20-2006 05:25 AM
Re: Searching Errors in a log file
./tlog: syntax error at line 25: `TOTAL_LINES=$' unexpected
There appears to be some syntax error in the line:-
TOTAL_LINES=$(awk '{nlines = nlines + 1}' END {print nlines} <$3)
=====================================================================
Below is the complete script.
script name:tlog
#!/usr/bin/sh
# Print number of lines, $2,
# after matched , case-insensitive string, $1,
# from file, $3
#
typeset -i FIRST_LINE=0 LAST_LINES=0 MATCH_LINE_NO=0 LINE_AFTER=0 TOTAL_LINES=0
# Check params..
if [ "$#" != 3 ]
then
echo wrong number of parameters
echo par1=string, par2=number of lines, par3=file
exit 1
fi
#
if [ ! -r $3 ]
then
echo Cannot read $3
exit 1
fi
# Number of lines in $3
TOTAL_LINES=$(awk '{nlines = nlines + 1}' END {print nlines} <$3)
grep -in "$1" "$3" | while read LINE
do
# Line number for match
MATCH_LINE_NO=$(echo $LINE | awk -F: '{print $1}')
# Check line number boundaries
FIRST_LINE=$(($MATCH_LINE_NO))
#
LAST_LINE=$(($MATCH_LINE_NO + $2))
if [ "$LAST_LINE" -gt "$TOTAL_LINES" ]
then
echo match in line $MATCH_LINE_NO, but cannot print lines after $TOTAL_LINES
fi
# Print after...
LINE_AFTER=$(($MATCH_LINE_NO + $2))
cat -n $3 | awk -v lineb=$FIRST_LINE -v linea=$LINE_AFTER '$1==lineb,$1==linea {print $0}'
echo
done
====================================================================
Appreciate suggestion.
Thanks,
Shiv
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2006 05:42 AM
12-20-2006 05:42 AM
Re: Searching Errors in a log file
from...
TOTAL_LINES=$(awk '{nlines = nlines + 1}' END {print nlines} <$3)
to...
TOTAL_LINES=$(awk '{nlines = nlines + 1} END {print nlines}' <$3)
~hope it helps
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2006 06:32 AM
12-20-2006 06:32 AM
Re: Searching Errors in a log file
now i am getting different error as:-
`MATCH_LINE_NO=$' unexpected
for the below line in the script:-
# Line number for match
MATCH_LINE_NO=$(echo $LINE | awk -F: '{print $1}')
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2006 07:12 AM
12-20-2006 07:12 AM
Re: Searching Errors in a log file
I'm not sure what you have. I copy-and-pasted John's script from your post of today (Dec 20). I corrected line-25 as Sandman pointed out and can run the script without errors and receive the output I expect.
I suggest you re-copy-and-paste and try the same.
Regards!
...JRF...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2006 08:04 AM
12-20-2006 08:04 AM
Re: Searching Errors in a log file
I copied John's script and it runs perfectly on my machine. As JRF says there is a problem with the copy 'n paste on your end.
~cheers
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2006 07:45 AM
12-21-2006 07:45 AM
Re: Searching Errors in a log file
Here is the messages:-
1st test:
$./t1 "Errors" 5 logfile
awk: record `java.class.path = /o...' too long
record number 1653
match in line 1732, but cannot print lines after 0
awk: syntax error near line 1
awk: bailing out near line 1
2nd test:
$./t1 "listening on port 1000" 5 logfile
match in line 2013, but cannot print lines after 0
awk: syntax error near line 1
awk: bailing out near line 1
Looks like some awk syntax errors ??
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2006 08:27 AM
12-21-2006 08:27 AM
Re: Searching Errors in a log file
From your output, I would guess that the log file has a record (a string of characters delimited with a newline) that exceeds an 'awk' internal limit. You might find, too, that you can't 'vi' the file for the same reason. Perl has no practical limits.
Regards!
...JRF...