Logs script help ..

[someone_4]

Hello,
I have a third part app. That creats daily log files named in such as AuditLog_2001-AUG-25.log
or DataPort2001_08_26.log even java_reports08282001.log. Now I have a script that rotates and clears logs such as syslog and mail.log. But It wont work for theese logs because there is not a log that theese apps are writting to. Theese are logs that already written at the end of the day. How would I go about writting a script that will rotate and compress the logs for 7 days and delete the old ones. Where I get stuck is writting a script that is smart enough to look for the files with differnt day formats. Thanks

Richard

[linuxfan]

Hi Richard,

There are quite a few scripts already available, pick your choice,

1. logrotate.sh
http://www.introcomp.co.uk/examples/logrotate.html

2. rotatelog program by shaun Rowland (in Perl)
http://www.interhack.net/projects/rotatelog
(pretty nice)

3. Rotate logs
http://www.ginini.com.au/tools/rotatelogs/

-Regards
Ramesh

[James R. Ferguson]

Hi Richard:

Let's ignore the format of the log's names.

Use the 'find' command to find logs that have been modified in the last 7-days, and pass their names onward to your compression script:

# find /logdir -type f ! -name "*.Z" -mtime -7 -exec $HOME/logzipper.sh {} \;

The "logzipper.sh" script would simply receive ( as $1) the name of the file found by the 'find' command and therein you could do anything with it you want.

Notice that I selected only files from the hypothetical /logdir and I skipped filenames ending in "Z" (which are already compressed).

Regards!

...JRF...

[Sridhar Bhaskarla]

Richard,

I tried to think of one way. If you know the directory where the log files are written, you can use this script.

For ex YOUR_LOG_DIR is your log directory where your log files are sitting. Create a backup directory under it.

#mkdir YOUR_LOG_DIR/backup

Now write a small script like this

cd YOUR_LOG_DIR
for i in 1 2 3 4 5 6
do
OLD=`echo $i + 1 |bc`

mv $YOUR_LOG_DIR/backup/AuditLog.${i} $YOUR_LOG_DIR/backup/AuditLog.${OLD}
done

mv $YOUR_LOG_DIR/AuditLog* $YOUR_LOG_DIR/Auditlog.1


Here in this case we are not preserving the formats. Simple way. If you need to preserve the formats, you need to make it bit complicated by duplicating the format using the date file.

-Sri

[linuxfan]

Hi Richard,

I missed the last part of your question.
Anyway, do all the log files contain .log as the extension? Also do all the log files contain the date irrespective of the format?

If so you could just look for files ending in *.log and containing todays date (date +%d)

In your script are you giving the names of the logs files individually?

-Ramesh

[Sridhar Bhaskarla]

Please change this

mv $YOUR_LOG_DIR/AuditLog* $YOUR_LOG_DIR/Auditlog.1

to

mv $YOUR_LOG_DIR/AuditLog* $YOUR_LOG_DIR/backup/Auditlog.1

-Sri

[someone_4]

Yes all the files end in a.log extension and the do contain the date irrespective to the format. I was thinking of doing -ot for older then but I couldnt get it to work. If I do ll
they do have
-rw------- 1 root sys 207922 Aug 30 17:03 java_reports08302001.l
og

So I was also thinking of doing a grep for the date and going from that. But I couldnt get that to work either. I have no idea where the log files are written I just know they end up in this dir.

Richard

[James R. Ferguson · Accepted solution]

Hi (again) Richard:

Based on your last post, go back to my first suggestion and amend it to look like:

# find /logdir -type f -name "*.log" -mtime -7 -exec $HOME/logzipper.sh {} \;

When you run this the second time, since the "logzipper.sh" was designed to compress files in the /logdir, the 'name' argument will *not* find "*.log.Z" files.

Regards!

...JRF...

[someone_4]

Mr James
when you mean logzipper.sh you mean my existing log clear script? If so here is the one I use. And I cant get it to work with the find command that you suggested



#!/usr/bin/ksh
#
# Cleanup logfiles (daily)
#
# Usage:
#
# dailylogs.sh [ DEBUG | TRACE ]
#
# where adding the word DEBUG (all caps) will
# confirm the logfile operations to stdout
# by showing each logfile's changes.
#
# The word TRACE will turn on shell tracing
# in all modules.
#
set -u

MYNAME=$(basename $0)
DEBUG=0
TRACE=0
if [ $# -gt 0 ]
then
case $1 in
TRACE)
set -x
TRACE=1
;;
DEBUG)
DEBUG=1
;;
esac
fi

#####################################################
function TrimLogFile
{
if [ $TRACE -eq 1 ]
then
set -x
fi

## Send over the full pathname of a logfile as MYLOGFILE
## Creates $MYARCHIVES archives and zeros the original file
## Does not rename the existing logfile since it is most likely
## open.
##
## Example: MYLOGFILE=/var/adm/syslog/syslog.log
## MYARCHIVES=5
## MYCHMOD=644
## MYCHOWN=root:sys
## Produces:
## /var/adm/syslog/syslog.log.5.Z
## /var/adm/syslog/syslog.log.4.Z
## /var/adm/syslog/syslog.log.3.Z
## /var/adm/syslog/syslog.log.2.Z
## /var/adm/syslog/syslog.log.1.Z
## /var/adm/syslog/syslog.log (zero length)
##
## If DEBUG=1 then report on the transactions

MYDIR=$(dirname $MYLOGFILE)
MYLOG=$(basename $MYLOGFILE)

# Go through the logs backwards starting with the oldest copy
# (defined by $MYARCHIVES)

NEXTLOG=$MYARCHIVES
while [ $NEXTLOG -gt 1 ]
do
PREVLOG=$(( NEXTLOG - 1 ))

# If the previous log exists (logs 1 through $MYARCHIVES-1) then
# remove the $NEXTLOG so as to preserve the date/time/permissions
# of the previous file. If the previous log does not exist, create
# a zero length file and set permissions per $MYCHMOD and $MYCHOWN.

if [ -f $MYDIR/$MYLOG.$PREVLOG.Z ]
then
rm -f $MYDIR/$MYLOG.$NEXTLOG.Z
cp -pf $MYDIR/$MYLOG.$PREVLOG.Z $MYDIR/$MYLOG.$NEXTLOG.Z 2>&1 > /dev/null
chmod $MYCHMOD $MYDIR/$MYLOG.$NEXTLOG.Z
chown $MYCHOWN $MYDIR/$MYLOG.$NEXTLOG.Z
else
touch $MYDIR/$MYLOG.$PREVLOG.Z
chmod $MYCHMOD $MYDIR/$MYLOG.$PREVLOG.Z
chown $MYCHOWN $MYDIR/$MYLOG.$PREVLOG.Z
fi
NEXTLOG=$(( $NEXTLOG - 1 ))
done

# Now take care of the current logfile

rm -f $MYDIR/$MYLOG.1
cp -pf $MYLOGFILE $MYDIR/$MYLOG.1
cat /dev/null > $MYLOGFILE
compress -f $MYDIR/$MYLOG.1
chmod $MYCHMOD $MYDIR/$MYLOG.1.Z
chown $MYCHOWN $MYDIR/$MYLOG.1.Z

if [ $DEBUG -eq 1 -o $TRACE -eq 1 ]
then
echo "Archive $MYARCHIVES copies and zero $MYLOGFILE:"
ll $MYLOGFILE
if [ -f $MYLOGFILE.*.Z ]
then
ll $MYLOGFILE*.Z
fi
fi
}

##################
## ##
## MAIN PROGRAM ##
## ##
##################

MYARCHIVES=7
MYCHOWN=root:sys
MYCHMOD=664
MYLOGFILE=/opt/java/simpletel/logs/AuditLog_200*
TrimLogFile

MYARCHIVES=7
MYCHOWN=root:sys
MYCHMOD=664
MYLOGFILE=/opt/java/simpletel/logs/DataPort*
TrimLogFile

MYARCHIVES=7
MYCHOWN=root:sys
MYCHMOD=664
MYLOGFILE=/opt/java/simpletel/logs/PINSpooler*
TrimLogFile

MYARCHIVES=7
MYCHOWN=root:sys
MYCHMOD=664
MYLOGFILE=/opt/java/simpletel/logs/java_invoices*
TrimLogFile

MYARCHIVES=7
MYCHOWN=root:sys
MYCHMOD=664
MYLOGFILE=/opt/java/simpletel/logs/java_reports*
TrimLogFile


####End of script
exit 0

[linuxfan]

Hi Richard,


I modified your script, try it.
Most of the changes were in the main program. Also modified it to use gzip rather than compress, because gzip does a much better job of compression.

-HTH
Ramesh

[James R. Ferguson]

Hi Richard:

Your 'dailylogs.sh' script handles one argument only -- DEBUG or TRACE. Moreover, the script expects fixed names for logs. Your request is to handle filenames with variable strings, namely dates.

I'm going to assume that your application periodically closes its log file and opens another one; hence the date portion of the filename.

Further, since the date string in the filename identifies the "rotation", I suggest that the whole process can be this:

#/usr/bin/sh
find /logdir -type f -name "*.log" -mtime +1 -exec compress -v {} \;
find /logdir -type f -name "*.log.Z" -mtime +7 -exec rm -i {} \;
exit 0
#.end

The first 'find' compresses logfiles named "*.log" in the /logdir but skips a log that's only a day old. The 'compress' does not change the modification timestamp. This is preserved for the ".Z" file.

The second 'find' removes compressed logs older than seven days. You probably want to drop the '-I' from the 'rm' after you have tested satistactorily.

Regards!

...JRF...

[someone_4]

Mr James

What is the differnce in
-mtime +7
-mtime -7
-mtime 7

I read the man pages but it didnt really clear thigns up for me.

[linuxfan]

Hi Richard,

-mtime +7 will work for files having been modified in more than 7 days
-mtime -7 for less than 7 days
-mtime 7 for exactly 7 days

-HTH
Ramesh

[James R. Ferguson]

Hi Richard:

From the man page for 'find', in the appropriate context, quote:

"In the descriptions of the primaries, the argument n represents a decimal integer; +n means more than n, -n means less than n, and n
means exactly n."

I'm not sure how better to phrase it. Hence '-mtime -7' would mean to find a file modified during the last 7-days.

I hope this helps!

Regards!

...JRF...