topic Re: reduce log size. in Operating System - HP-UX

reduce log size.

mick001 — Thu, 24 Jun 2004 05:28:18 GMT

Hi All,

I have a large file log 150 MB of text and I like to remove all history except for the last 30 dayâ s
The number 149 indicates that a new day is started.

How can I do this quick?

MANY THANKS

Mi

Re: reduce log size.

Geoff Wild — Thu, 24 Jun 2004 05:37:30 GMT

You could try something like:

tail -1000 log > newlog

cat /dev/null >log

cat newlog > log

Change the 1000 to where ever you want to start from.....

Rgds...Geoff

Re: reduce log size.

Mobeen_1 — Thu, 24 Jun 2004 05:38:00 GMT

Mi,
I think the simplest and the best way to do this is by searching for that file for a time stamp and output the result to another file.

regards
Mobeen

Re: reduce log size.

Sanjay Kumar Suri — Thu, 24 Jun 2004 05:38:48 GMT

Find out the line postion from where you want to Keep and use the following

tail -line_number logfile >> logfile.new
mv logfile.new logfile

sks

Re: reduce log size.

mick001 — Thu, 24 Jun 2004 05:41:53 GMT

Hi All,

If I use tail -1000 log >log.new
wc -l log.new I only get 137 lines.

The log file have to start with 149...

THANKS

Re: reduce log size.

Manish Srivastava — Thu, 24 Jun 2004 05:44:03 GMT

Hi,

You can do something like :

grep -n 149 file | tail -16

Do a wc -l on the output to make sure that there are more than 15 days data by
if wc -l > 15

then do a head -n 1 to get the beginning of the record of 15th day subtract 1 from it.

To get the no. of lines to be deleted do

cut -f1 -d":"

do wc -l on the whole file - the no. of lines to be deleted, do a til of these many lines and store it in a file.

I know i have made it a bit confusing but this is the best I could do as I cannot write a script right now.

manish.

Re: reduce log size.

Simon Hargrave — Thu, 24 Jun 2004 05:45:28 GMT

Something like this: -

LINE=`grep -n "^149" logfile| tail -30 | head -1 | cut -d: -f1`

This will set the var $LINE to be the line number of the n-30'th day's start.

Then do: -

awk 'NR>'$LINE'' logfile > newlogfile
mv newlogfile logfile

If you want to do it inline, you could probably use an "ed" script to edit the file, once you have the line numbe from the first command.

Sy

Re: reduce log size.

Fred Ruffet — Thu, 24 Jun 2004 05:59:20 GMT

$ cat file
hello
world
149
foo
bar
149
what
else
can
I
put
here
149
?

$ cat run
#!/opt/perl/bin/perl

$Index=0;
$MaxDays=2;
$Separator="149";
while () {
chomp;
$CurrentLine=$_;
if ($CurrentLine eq $Separator) {
$Index=($Index+1)%$MaxDays;
$MyTab[$Index]="";
} else {
$MyTab[$Index].=$CurrentLine."\n";
}
}
for ($Count=($Index+1)%$MaxDays;$Count<=$Index;$Count++) {
print $MyTab[$Count];
if ($Count<$Index) { print $Separator."\n"; }
}

$ ./run < file
what
else
can
I
put
here
149
?

There is surely better, but it works.

Regards,

Fred

Re: reduce log size.

mick001 — Thu, 24 Jun 2004 07:00:05 GMT

Hi All,

I try the commands from Manish Srivastava.
But then is stil have one problem.
The command AWK remove the fist 149 line!
This must be the first line other wise it dousn't work.

THANKS

Re: reduce log size.

Simon Hargrave — Thu, 24 Jun 2004 07:48:30 GMT

If you do it with the awk as per my answer, then you can easily change this.

eg it says 'NF>'$LINES'', but it could also say 'NF>'$LINES'-1' etc, to tune to your exact requirements.

Re: reduce log size.

Nicolas Dumeige — Thu, 24 Jun 2004 10:15:24 GMT

Hello,

If the log is in use while you're purging it, this can be trouble.

- If you create a new file with your n lines and move it, the file descriptor of the application wihc create the log will be pointing at the wrong file, it will still be able to write - as the unlink remove only the reference to the file - but what will become of the line written ?

- if you work on the same file, issuing a cat /dev/null > log ; cat n_lines_log > log ; you'll get a sparse file.

My 2 cents

Cheers

Nicolas

Re: reduce log size.

Nicolas Dumeige — Thu, 24 Jun 2004 10:17:47 GMT

Have you pick the editor option ?
Opening a 150 Mo file can be too much if you don't have enought resources.

Re: reduce log size.

Nicolas Dumeige — Thu, 24 Jun 2004 10:21:40 GMT

Won't be very fast as well !