topic Re: Tail alternative in Operating System - HP-UX

Tail alternative

dude70_1 — Wed, 17 Dec 2003 11:00:03 GMT

Hi Guys!

I am trying to change the monitoring ksh script into perl script. Can this be converted in perl script? Also Is it possible to mix perl script and shell script as one script?

**********
tail -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************

The reason being tail -f sleeps for one second according to the UNIX man pages! Sometimes the log file may be appended with lots of lines So tail is doing it very slow. It puts things in buffer and works very slow on that. I heard perl is a best alternative!

Thanks.

Re: Tail alternative

Michael Schulte zur Sur — Wed, 17 Dec 2003 11:13:31 GMT

Hi,

that shouldnt matter. Most important thing is, that it doesnt miss any data. Do you have proof, that it misses data? You can of course put perl commands into the ksh script.

greetings,

Michael

Re: Tail alternative

dude70_1 — Wed, 17 Dec 2003 11:29:03 GMT

Hi Mike,

Thanks for the reply. I don't see that missing any data. My log files are dumped with enormous data. After determining the error I have to do lots of operations. So I am wondering any other thing will improve the speed!

Thanks again!

Re: Tail alternative

Paula J Frazer-Campbell — Wed, 17 Dec 2003 11:34:13 GMT

Hi

In this use tail is just looking at the last 10 lines once each second so increase the number of lines that it looks at until you feel that you are looking far enough back into the file to catch all errors.

**********
tail -n NN -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************
NN = number of lines to look at

so for 50 lines:-

**********
tail -n 50 -f log.txt | while read LINE
do
var= `echo $LINE | grep "Error Occured"`
echo $var
done
**************

Paula

Re: Tail alternative

Michael Schulte zur Sur — Wed, 17 Dec 2003 11:38:41 GMT

Hi Paula,

if there are so many data written into the log, your approach will miss information, plus you will have to run it each time new.

Hi Dude70,

can you make a more precise search for the grep, so you get less lines?

greetings,

Michael

Re: Tail alternative

Paula J Frazer-Campbell — Wed, 17 Dec 2003 11:41:12 GMT

Hi

An option is the grep out all errors and store them in an error log and then use the tail routine on that.

Collect so:-

**********
cat log.txt | while read LINE
do
echo $LINE | grep "Error Occured" > errorlog.txt
done
**************

View so:-

then either cat or tail -n NN errorlog.txt

Paula

Re: Tail alternative

Jean-Louis Phelix — Wed, 17 Dec 2003 11:55:05 GMT

Hi,

I'm not sure that tail if the problem ... It goes in the 1s loop ONLY if there is not data available, so in case of many lines comming suddenly they will be immediatly displayed. But perhaps it's the application which bufferize its output ? Sometimes a fflush() in a C program can improve ...

Regards.

Re: Tail alternative

dude70_1 — Wed, 17 Dec 2003 11:55:33 GMT

Hi Mike,

I have lots of data apart from "Error Occured" string written to the log. I have to look for this particular string. If this happens then I have to add following 5 lines which describes the error data and process them. Paula's approach will be good if I have only one line/one file. In contrast I have zillon files and too much of data! I was wondering whether perl will do a better job. Now I am also working for C and java alternatives!

Thanks.

Re: Tail alternative

Michael Schulte zur Sur — Wed, 17 Dec 2003 12:00:23 GMT

Hi,

try this:
tail -f log.txt | while read LINE
do
if test `echo $LINE | grep "Error Occured" | wc -l` -gt 0
then
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
read LINE
echo ${LINE}
done

greetings,

Michael

Re: Tail alternative

Michael Schulte zur Sur — Wed, 17 Dec 2003 12:01:58 GMT

Hi,

forgot the fi before the done!

Michael

Re: Tail alternative

Graham Cameron_1 — Wed, 17 Dec 2003 12:08:38 GMT

tail -f log.txt | awk '/Error Occured/ {print; for (i=1;i<=4;i++) {getline;print}} '

-- Graham

Re: Tail alternative

Paula J Frazer-Campbell — Wed, 17 Dec 2003 12:11:23 GMT

Hi

Have a t look at swatch:-

http://ciac.llnl.gov/ciac/ToolsUnixSysMon.html#Swatch

Paula

Re: Tail alternative

dude70_1 — Wed, 17 Dec 2003 12:27:15 GMT

Thanks guys,

I'll post the results after testign these!

Cheers!

Re: Tail alternative

dude70_1 — Wed, 17 Dec 2003 12:36:41 GMT

Hi Mike and Graham,

Can you explain what exactly your code does?

Thanks!

Re: Tail alternative

Michael Schulte zur Sur — Wed, 17 Dec 2003 16:35:51 GMT

Hi Dude70,

my script is looking for the error text "Error Occured" prints it and the next five lines following it, as you requested.

greetings,

michael

Re: Tail alternative

Graham Cameron_1 — Thu, 18 Dec 2003 03:16:20 GMT

Dude

My contribution is in 2 parts.

The tail -f part I think you know already.
The awk script does nothing until it finds the string "Error Occured" (BTW - is this spelled correctly).
Then the solitary "print" prints the current line.
Then the for loop reads in 4 more lines and prints them.
Then it waits for another occurence of the string.

The general format of an awk script is a collection of
pattern { action }
statements, so in this case pattern is "Error Occured" and action is everything between the outside braces.

Hope this helps.

-- Graham

Re: Tail alternative

Mark Grant — Thu, 18 Dec 2003 03:51:24 GMT

The problem here is that if you don't do what "tail" does you could potentially cripple the machine. You have to sleep!

The alternative is for your script to open syslog and read from it, If there is nothing to read, it will hang until there is somethign else to read. This sounds great but will kill the performance of the machine.

Try this to see what I mean.

#!/usr/bin/perl

open SYSLOG,"
while (1){
$i=;
if($i=~/Error Occured/){
print "Oh dear, we got another error $i";

}
}

Re: Tail alternative

dude70_1 — Fri, 19 Dec 2003 16:49:11 GMT

Thanks guys!

It works better now.