reduce log size.

[mick001]

Hi All,

I have a large file log 150 MB of text and I like to remove all history except for the last 30 dayâ s
The number 149 indicates that a new day is started.

How can I do this quick?

MANY THANKS

Mi

[Geoff Wild]

You could try something like:

tail -1000 log > newlog

cat /dev/null >log

cat newlog > log


Change the 1000 to where ever you want to start from.....

Rgds...Geoff

[Mobeen_1]

Mi,
I think the simplest and the best way to do this is by searching for that file for a time stamp and output the result to another file.

regards
Mobeen

[Sanjay Kumar Suri]

Find out the line postion from where you want to Keep and use the following

tail -line_number logfile >> logfile.new
mv logfile.new logfile

sks

[mick001]

Hi All,

If I use tail -1000 log >log.new
wc -l log.new I only get 137 lines.

The log file have to start with 149...

THANKS

[Manish Srivastava]

Hi,

You can do something like :

grep -n 149 file | tail -16

Do a wc -l on the output to make sure that there are more than 15 days data by
if wc -l > 15

then do a head -n 1 to get the beginning of the record of 15th day subtract 1 from it.

To get the no. of lines to be deleted do

cut -f1 -d":"

do wc -l on the whole file - the no. of lines to be deleted, do a til of these many lines and store it in a file.

I know i have made it a bit confusing but this is the best I could do as I cannot write a script right now.

manish.

[Simon Hargrave · Accepted solution]

Something like this: -

LINE=`grep -n "^149" logfile| tail -30 | head -1 | cut -d: -f1`

This will set the var $LINE to be the line number of the n-30'th day's start.

Then do: -

awk 'NR>'$LINE'' logfile > newlogfile
mv newlogfile logfile

If you want to do it inline, you could probably use an "ed" script to edit the file, once you have the line numbe from the first command.


Sy

[Fred Ruffet]

$ cat file
hello
world
149
foo
bar
149
what
else
can
I
put
here
149
?

$ cat run
#!/opt/perl/bin/perl

$Index=0;
$MaxDays=2;
$Separator="149";
while () {
chomp;
$CurrentLine=$_;
if ($CurrentLine eq $Separator) {
$Index=($Index+1)%$MaxDays;
$MyTab[$Index]="";
} else {
$MyTab[$Index].=$CurrentLine."\n";
}
}
for ($Count=($Index+1)%$MaxDays;$Count<=$Index;$Count++) {
print $MyTab[$Count];
if ($Count<$Index) { print $Separator."\n"; }
}

$ ./run < file
what
else
can
I
put
here
149
?

There is surely better, but it works.

Regards,

Fred

[mick001]

Hi All,

I try the commands from Manish Srivastava.
But then is stil have one problem.
The command AWK remove the fist 149 line!
This must be the first line other wise it dousn't work.

THANKS

[Simon Hargrave]

If you do it with the awk as per my answer, then you can easily change this.

eg it says 'NF>'$LINES'', but it could also say 'NF>'$LINES'-1' etc, to tune to your exact requirements.

[Nicolas Dumeige]

Hello,

If the log is in use while you're purging it, this can be trouble.

- If you create a new file with your n lines and move it, the file descriptor of the application wihc create the log will be pointing at the wrong file, it will still be able to write - as the unlink remove only the reference to the file - but what will become of the line written ?

- if you work on the same file, issuing a cat /dev/null > log ; cat n_lines_log > log ; you'll get a sparse file.

My 2 cents

Cheers

Nicolas

[Nicolas Dumeige]

Have you pick the editor option ?
Opening a 150 Mo file can be too much if you don't have enought resources.

[Nicolas Dumeige]

Won't be very fast as well !