Trim a log file by date

[Norman England]

Okay here goes...

I have scripts that run on a box that need to have log files for the past two months. However the files do get quite large, and I need to trim them to only include the pat 60 days. Here is an example of the output of the log file:

Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 RECESS Not Valid.
Mon Feb 11 15:07:34 EST 2002 Bulding DataLog File test_A87026.1_I9-REC3-C_01
Mon Feb 11 15:07:34 EST 2002 ./RECZ01EA.tcl: RECZ01EA A87026.1 I9-REC3-C Finished.


My question is how do I create the script to only remove entires older than 60 days?

There could be any number of entries ranging from none in a day, to 100 in a day, so just tailing the end of the file won't work...

thanks for the help.

[S.K. Chan]

Just curious, would it be better to have a process in place to archive and compress the "full" log file to somewhere else vs deleting the entries in it ? Those archived log files can be deleted on a regular basis after.

[A. Clay Stephenson]

Hi Norman:

This should be fairly close. The idea is to parse that date format into a numerical month, day, year format and then use that to feed me universal date hammer 'caljd.sh'. It returns the number of days since ~Jan 1, 4713BCE (the Julian Day). If we compare the today's Julian Day to that of your log entry and the age is greater than 60 we simply don't echo the line.

I'll do this in two attachments so that you get both parts:

The general idea is this:

#!/usr/bin/sh
TDIR=${TMPDIR:-/var/tmp}
PID=$$
TF1=${TDIR}/X${PID}_1.txt
LOGFILE=mylogfile

cat ${LOGFILE} | awk -f trim.awk > ${TF1}
STAT=$?
if [ ${STAT} -eq 0 ]
then
mv ${TF1} ${LOGFILE}
else
rm -f ${TF1}
fi
exit ${STAT}


Here is the first part, 'trim.awk'. If you like, you could make it a part of the above script with a 'here doc'.

Regards, Clay

[A. Clay Stephenson]

Hi again:

And now the other part, caljd.sh.

[harry d brown jr]

Norman, it sounds like it would be easier to have the logs "rotated" once a month, or maybe even "weekly".

live free or die
harry