How to truncate Log file to contain one month data only

[Robert Peregrin]

Hi,

I have a log file which is growing to almost 2 GB in size. I need to truncate this log file to contain only one month old data in order to reduce its size.

Please help in giving a procedure for this.


Regards,

Robert Peregrin

[Suraj K Sankari]

Hi,

to truncate you 0 you just need to give a > infront of your file.
>filename

to truncate file with condition then you need to write a script.

Suraj

[Dennis Handly]

Which logfile? Some are ascii text (syslog.log) and some are binary (wtmps).

It also depends on what is the date string in the file. And whether you need exactly 31 days or just the current month or you are only going to run the script on the last/first day of the month.

[Emil Velez]

in SAM or the SMH tools there is a section under routine tasks where you can automatically trim log files and you can add non-standard log files to this tool and define a max size for the log file. YOu can then trim all of them at the same time and periodically using this tool.

[Raj D.]

Robert,

If you want to trim the file , by reducing contents from the log file you can do so by removing first few lines,
1. count the lines, with wc -l logfile ,
2. trim it as desired.

- If you want a gzip copy of backup of the file elsewhere you can keep a copy.

-Example: To trim first 2000 lines from the file:
- stop the process writing the file.
- trim
# perl -i -ne 'print unless 1 .. 2000' logfile.log
- start the process, writing to it.
- check size/lines.

Hth,
Raj.

[rariasn]

Hi Robert:

Create a script to switch and compress the log file.

- Stop application
- mv logfile to logfile.old
- Initialize new logfile
- Start application
- compress the old logfile

schedule (cron, ovo, ...) the script in specified dates and times.

rgs,

[Robert Peregrin]

Thanks for your suggestions guys.

The log file is a text file.

The SAM utility is nice but we need the triming to be automatic. A simple script would be useful which I can run as a cron job.

We are currently using the below script. If you can have better and easier suggestions, then it would be helpful.

#!/usr/bin/sh
#
# this script should be run (via cron) daily to keep DMC log at a
# reasonable size
#
WHOAMI=/usr/bin/whoami
ECHO=/usr/bin/echo
CP=/usr/bin/cp
DATE=/usr/bin/date
CAT=/usr/bin/cat
ED=/usr/bin/ed
LOGGER=/usr/bin/logger
DMCLOGDIR=/var/opt/dmc/logs
DMCLOG=dmc.log

#
# make sure script is running as root
#
if [ `$WHOAMI` != "root" ] ; then
$ECHO you must be root to run this script
exit
fi

#
# copy the DMC log file
#
cd $DMCLOGDIR
$CP $DMCLOG ${DMCLOG}.Day`$DATE -u +%d`

#
#
# truncate the DMC log file
#
$CAT - <1,$ d
w
q
EOF

#
# put log entry in the syslog file
#
$LOGGER $DMCLOG truncated at `$DATE`

[rariasn]

Hi Robert:

#
#
# truncate the DMC log file
#

> $DMCLOG

rgs,

[Robert Peregrin]

Note: I need to truncate the log file but it should still contain 1 month old data.

[rariasn]

Hi Rorbert:

Sample scheduled script with cron:

# cd /var/spool/cron/crontabs

# cat root

30 1 1 * * /tmp/test.sh #Test

# ll root

-rw------- 1 root sys 30 Aug 16 12:43 root

The entries in a crontab file are lines of six fields each. The fields are separated by spaces or tabs. The lines have the following format:

minute hour monthday month weekday command


# crontab -l



rgs,

[Dennis Handly]

>it should still contain 1 month old data.

What is the format of the file and what are the answers to my other questions?

[Robert Peregrin]

The file is called dmc.log. Its an application specific log file in text format.

Sample date string in the file is "Sat Aug 14 18:42:19 2010."

Yes, I need to keep exactly 1 month old data (number of days depends on the month as some months have 30 days, some have 31 and February have 29 or 28 days) in the file.

It doesn't matter when I run the script for as long as I will be able to truncate the file with 1 month old data kept.

[Elmar P. Kolkman]

This should work for selecting the lines from the log file.

#!/usr/bin/perl -w

use strict;

my @prev_months= qw(Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);

my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);

my $fh;
open($fh,$ARGV[0]);
my $line;
my @parts;

while ($line = <$fh>) {
@parts=split(' +',$line);
if (($parts[1] eq $prev_months[$mon]) && ($parts[2] > $mday)) {
print $line;
}
if (($parts[1] eq $prev_months[($mon+1)]) && ($parts[2] <= $mday)) {
print $line;
}
}

[Dennis Handly]

>I need to keep exactly 1 month old data (number of days depends on the month as some months have 30 days, some have 31 and February have 29 or 28 days) in the file.

The number of days in a month is useless unless you are at the end of the month. If you are in the middle, what does it mean?

>It doesn't matter when I run the script for as long as I will be able to truncate the file with 1 month old data kept.

I assume we can make the assumption that if today is Aug 16, you want the info from July 16 to today?

You could make an awk script to scan every record for these fields:
Sat Aug 14 18:42:19 2010

if ($2 == prevmon && $3 >= currday && $5 >= prevyear ||
$2 == currmon && $5 == curryear) copy=1

You would have to do date arithmetic to figure out what the previous month's name and year is. The current month, day and year can be gotten from date(1).

Once "copy" is set, just read & print to the end of the file.

[Robert Peregrin]

Thanks Guys. Your suggestions are good but we chose to retain the running script we are using.

[Robert Peregrin]

I have already found an alternative solution to my question.