Direct trimming of a logfile

[David Yandry]

Hello Experts,

I have an unusual request. I have a very large logfile that I would like to trim each day. I would like to remove perhaps the first 5000 lines and keep the rest of the file. I do not want to create a new file but rather edit this logfile directly using a script. Any ideas?

Thanks,
David

[Steven E. Protter]

Here is how I'd do it.

tail 10000 logfilename > newlogname
cp logfilename /tmp/logfilearchive
mv newlogname logfilename

Let cron do it once a day.

This insures you get the last 10000 lines of the log file in question every day. It is an effective trim.

SEP

[G. Vrijhoeven]

Hi David,

You could use cat $LOG head -n 5000 >> $LOG2 && mv LOG2 LOG ,

But this will create a new file.

Gideon

[A. Clay Stephenson · Accepted solution]

This isn't too difficult. We can leverage Perl's truncate() function to chop off the end of a file on UNIX's that support the truncate() system call --- and HP-UX does.

The trick is to open two filehandles on the same pathname: one for input and the other for update. We read off the first lines and then when the first line to keep is found we start copying to the update filehandle. When done, we use the tell() function to determine the offset for the last write operation and truncate the file at that point. It's probably taken me longer to 'splain this than it did to write the Perl code using a bit of copy and paste.

Here's trunc.pl; Use it like trunc.pl mylogfile 5000 to throw away the 1st 5000 lines of mylogfile.

[Omololu Shobayo]

is the file an ASCII logfile?

[Jeff Schussele]

I don't believe tail is gonna fly in this scenario. It works with only a 20KB buffer, so I seriously doubt it would work on a 5000+ line logifile - unless they were very small lines.
I don't like to use tail for more than 100-200 lines due to this limitation.

My 2 cents,
Jeff

[A. Clay Stephenson]

I think you can assume that when the word "lines" is used, it is, be definition, text. ASCII encoding is also all but a sure thing.

[David Yandry]

Hello,

Yes, the logfile is a large ASCII text file.

Thanks for all the ideas so far. I especially like A. Clay's approach. Clay, would it be possible to capture the first lines in another file instead of just ingoring them them?

Thanks,
David

[Omololu Shobayo]

I would have suggested splitting the file using split command but, you may have a size smaller than required at the end of the split. can we start splitting from the back.

i.e a file of size 200K splitted in 15ks would have 13 of 15K and 1 of 5K. The last 1K would be the last but if we split from the end the 5K would be the 1st.

Let me check.

[Geoff Wild]

Or try logrotate:

"Logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large."

http://hpux.ee.ualberta.ca/hppd/hpux/Sysadmin/logrotate-2.5/

Rgds...Geoff

[Steven E. Protter]

Thanks Jeff,

I'll give it another hack then.

#!/usr/contrib/bin/perl

$Argument = ($ARGV[0]);
$instance = $Argument;


$countfile = "/var/adm/syslog.log";
$newlog = "/var/adm/syslog.log.new";
open (COUNT, "$countfile");
open (NEWLOG, "$newlog");

# print "Made it here bud!!!!\n";

$n1 = 0;


@filedata = ;
foreach $filedata (@filedata){
chop ($filedata) if ($filedata =~/\n$/);
# print "$filedata eol \n";
$filetoback = $filedata;
$n1 = $n1 + 1;
print NEWLOG "$filetoback";
if ($count ge 50000) {
close(COUNT);
close(NEWLOG);
exit;

}
}



The chop command takes off carraige return line feed. You probably don't want to do that.

You will have to adjustments to get the LAST 50000 lines of the log.

SEP

[Omololu Shobayo]

yes you can split by line count. Try to manupulate the split command and you will get it.

[A. Clay Stephenson]

Sure, that's quite "doable". Perhaps, the easist method is to simply add a
"print $s;" line just before the "++$knt;" line in the first while loop. That will send the lines to be discarded to stdout. In the case where those lines truly don't need to be kept then simply redirect stdout to /dev/null.

[Omololu Shobayo]

get the length of the logfile from wc -l
LINE= `wc -l logfile

get the value minus 500 lines= NEWLINE

then split using the -l of the value of NEWLINE.

this will create 2 files one would be very big and the other will be what you need.

delete what you do not need and rename the fiel of size 500 lines.

You may need to add some logics as you go along. This is simple for any level

[David Yandry]

Thanks everyone. A. Clay's solution wins the prize. Why am I not surprised? Unlike the other ideas, his works directly on the file and does not require any temp files.

Thanks to all.

David