ideas on how to archive a log that cant be cleared...

[someone_4]

Hello everyone

I am running into a jam here.
I have a log lets call it log.log for an application that is running 24/7. Of course the log logs live.

Now the issue is that I need to archive the log every 1 - 4 hours. But I can not clear out the live log.

Here is what i was running everyhour.

for SWITCHID in 1 2
do
cd /var/apps/log
cd $SWITCHID

nice -n 3 tar -cvf - log.log |nice -n 3 gzip | nice -n 3 remsh lv_a
rchive -l archive "cat > /archive/dbsvr1/$SWITCHID/logs-`date +%Y%m%d.%H%M`.tar.
gz"
sed 'd' log.log > log.log
cd ..
done


But it clears out log.log every hour. And I was asked not to clear out log.log but archive every hour.

Is there a good way to do this?

Thanks
Richard

[Jeff_Traigle]

My first thought is to simply remove the sed command that clears the log. However, I feel I must be misunderstanding the requirement you stated since that's exceedingly simple. :)

If the requirement is to just archive the additional entries in that log in the hour since the last archive without clearing it, then you could probably use the diff command for that. Have an offline copy of the log from the previous hour, make copy of active log (to avoid it changing between comparison and copy to offline version), do diff to the hourly archive, then rotate the offline logs to use in comparison the next hour.

Is that close to what you're trying to accomplish?

[Gary L. Paveza, Jr.]

Why not make a copy of the log, then clear it:

ARCHIVE_TIME=$(date +%C%y%m%d%H%M)

cp logfile logfile.${ARCHIVE_TIME}cat /dev/null > logfile

You may want to compress the archive.

either
gzip logfile.${ARCHIVE_TIME}
or
compress logfile.${ARCHIVE_TIME}

[Dave La Mar]

Richard -
If the log file is write intensive it will be difficult not to miss data.
As noted
1. Make a copy.
then
2. wc the copy and only delete those lines 1,X
from the active file.

Hope to see a better solution here.

Best of luck.

regards,

dl

[curt larson_1]

here is another way

#checkpoint the file
print "Checkpoint for archiving ..." >> log.log

#now you want to archive the data between the
#last two checkpoints

#use awk, grep, perl to get the line numbers
#of the last two checkpoints

firstCheck=#checkpoint 2nd to the end of log.log
lastCheck=#last checkpoint in file

#if you don't want the checkpoint lines
firstCheck= firstCheck +1
lastCheck=lastCheck -1

#then use awk, sed, perl to get those line
# to your archive

sed "firstCheck,lastCheck p" > archive.log

I haven't done all the details put I'm sure it won't be to hard to complete if your interesting in using this method

myself I won't suggest keeping a log file without ever clearing it. if the application stays up long enough your going to fill up your disk space. (I hope your log is on a seperate file system from /var) and as the file size increases your performance is going to decrease. just how much is hard to say.

i'd suggest doing something like this

#copy log file
cp log.log log.bck

#do your archiving, compressing, backup to tape, etc

#if all that completed successfully

#get number of lines in log.bck
numLines=$(awk 'END {print NR;}')

#use ex, perl to to delete numLines from the begining of log.log

ex -s +"$numLines dd | wq!" log.log

rm log.bck

[Elmar P. Kolkman]

Or something like this. Its not fool-proof, but should do what you want, archive the logging done in the last timespan. Mind that archive can not be gzipped this way.

NUMLINES=$(cat log.log | wc -l)
OLDNL=0$(cat lastcheckpoint)
if [ $NUMLINES -gt $OLDNL ]
then
awk -v NL=$NUMLINES -v ONL=$OLDNL '
NR>ONL && NR <= NUMLINES { print }' log.log >>archive
elif [ $NUMLINES -eq $OLDNL ]
then
: # No new logging
else
head -$NUMLINES log.log >> archive
fi

echo $NUMLINES > lastcheckpoint