Tail alternative

[dude70_1]

Hi Guys!

I am trying to change the monitoring ksh script into perl script. Can this be converted in perl script? Also Is it possible to mix perl script and shell script as one script?


**********
tail -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************


The reason being tail -f sleeps for one second according to the UNIX man pages! Sometimes the log file may be appended with lots of lines So tail is doing it very slow. It puts things in buffer and works very slow on that. I heard perl is a best alternative!

Thanks.

[Michael Schulte zur Sur]

Hi,

that shouldnt matter. Most important thing is, that it doesnt miss any data. Do you have proof, that it misses data? You can of course put perl commands into the ksh script.

greetings,

Michael

[dude70_1]

Hi Mike,

Thanks for the reply. I don't see that missing any data. My log files are dumped with enormous data. After determining the error I have to do lots of operations. So I am wondering any other thing will improve the speed!

Thanks again!

[Paula J Frazer-Campbell]

Hi

In this use tail is just looking at the last 10 lines once each second so increase the number of lines that it looks at until you feel that you are looking far enough back into the file to catch all errors.


**********
tail -n NN -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************
NN = number of lines to look at

so for 50 lines:-

**********
tail -n 50 -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************


Paula

[Michael Schulte zur Sur]

Hi Paula,

if there are so many data written into the log, your approach will miss information, plus you will have to run it each time new.

Hi Dude70,

can you make a more precise search for the grep, so you get less lines?

greetings,

Michael

[Paula J Frazer-Campbell]

Hi

An option is the grep out all errors and store them in an error log and then use the tail routine on that.


Collect so:-

**********
cat log.txt | while read LINE
do
echo $LINE | grep "Error Occured" > errorlog.txt
done
**************

View so:-

then either cat or tail -n NN errorlog.txt


Paula

[Jean-Louis Phelix]

Hi,

I'm not sure that tail if the problem ... It goes in the 1s loop ONLY if there is not data available, so in case of many lines comming suddenly they will be immediatly displayed. But perhaps it's the application which bufferize its output ? Sometimes a fflush() in a C program can improve ...

Regards.

[dude70_1]

Hi Mike,

I have lots of data apart from "Error Occured" string written to the log. I have to look for this particular string. If this happens then I have to add following 5 lines which describes the error data and process them. Paula's approach will be good if I have only one line/one file. In contrast I have zillon files and too much of data! I was wondering whether perl will do a better job. Now I am also working for C and java alternatives!

Thanks.

[Michael Schulte zur Sur]

Hi,

try this:
tail -f log.txt | while read LINE
do
if test `echo $LINE | grep "Error Occured" | wc -l` -gt 0
then
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
done


greetings,

Michael

[Michael Schulte zur Sur]

Hi,

forgot the fi before the done!

Michael

[Graham Cameron_1]

tail -f log.txt | awk '/Error Occured/ {print; for (i=1;i<=4;i++) {getline;print}} '

-- Graham

[Paula J Frazer-Campbell]

Hi

Have a t look at swatch:-

http://ciac.llnl.gov/ciac/ToolsUnixSysMon.html#Swatch


Paula

[dude70_1]

Thanks guys,

I'll post the results after testign these!

Cheers!

[dude70_1]

Hi Mike and Graham,

Can you explain what exactly your code does?

Thanks!

[Michael Schulte zur Sur · Accepted solution]

Hi Dude70,

my script is looking for the error text "Error Occured" prints it and the next five lines following it, as you requested.

greetings,

michael

[Graham Cameron_1]

Dude

My contribution is in 2 parts.

The tail -f part I think you know already.
The awk script does nothing until it finds the string "Error Occured" (BTW - is this spelled correctly).
Then the solitary "print" prints the current line.
Then the for loop reads in 4 more lines and prints them.
Then it waits for another occurence of the string.

The general format of an awk script is a collection of
pattern { action }
statements, so in this case pattern is "Error Occured" and action is everything between the outside braces.

Hope this helps.

-- Graham

[Mark Grant]

The problem here is that if you don't do what "tail" does you could potentially cripple the machine. You have to sleep!

The alternative is for your script to open syslog and read from it, If there is nothing to read, it will hang until there is somethign else to read. This sounds great but will kill the performance of the machine.

Try this to see what I mean.

#!/usr/bin/perl

open SYSLOG,"
while (1){
$i=;
if($i=~/Error Occured/){
print "Oh dear, we got another error $i";

}
}

[dude70_1]

Thanks guys!

It works better now.