syslog monitoring and size control

[Bill McNAMARA_1]

Hi, just had a full file system there because of an enormous syslog.

I'm looking for a way (other than cron) to configure the syslog max size, log roll over options.. I couldn't find anything appropriate in syslogd or syslog.conf

Any ideas,
Later,
Bill

[Stefan Farrelly]

I dont believe there is a proper way to control the size of syslog.
We use a job which checks for this size of syslog, over a certain size and cp /dev/null to it. Works fine.

[James R. Ferguson]

Hi Bill:

I'm curious why you want to avoid 'cron' (or an 'at' job refeeding itself) to periodically manage trimming.

The other thought that comes to mind is to split up the syslog information into other log files (as configured in /etc/syslog.conf) and mount those as discrete filesystems. Perhaps that would faciliate your management/tracking tasks.

Regards!

...JRF...

[Volker Borowski]

Hmmm,

usally, I would like to read the syslog esp. then, when there is a lot of stuff in it, because then something is wrong. Dumping it away would be no good in my opinion (beside doing it manually after a check, which would be OK).

/etc/syslog.conf could be a solution. You could reduce the loglevel for some events and configure mail notification for critical events to users which are more frequently logged in than root.

I think, if you have a system which is so chatty in the syslog, that it fills up a filesystem, then this system really needs help, otherwise it would not complain in this way.
If you think of some information, not beining important, configure them inactive or redirected to /dev/null in syslog.conf.

On NT for example, you can configure a limit for the eventlog. I had it more than once, on NT, when a disk went bad, that 5000 entries rolled around the FIRST message within 10 minutes. That erased the information when the problem did start (Whicht would be the information, I would need for recovery purposes).

Sorry Bill, in my opinion, a self-reorganizing syslog will lead to deleted information some day. I would not like to have it this way (See NT above).

Have a nice cronjob, archiving your syslogs to a place where enough space is. On day you will need it.

Just my 2 ?
Volker

[linuxfan]

Hi Bill,

There is an ascii file /usr/sam/lib/rt/logs.list which lists a bunch of log files and the recommended size for them. Also if you don't want to set up cron then within SAM in the "Routing Tasks" there is another subsection for "System Log Files" where you can trim individual log files to a "zero" to the "recommended size" or "to a certain percentage". In case you want to change the recommeded size you can do it, within SAM or by editing the ascii file.

But there are quite a few tools available which make your life easier, and if you want to take a look at them, here are a few of them

1. logrotate.sh
http://www.introcomp.co.uk/examples/logrotate.html

2. rotatelog program by shaun Rowland (in Perl)
http://www.interhack.net/projects/rotatelog
(pretty nice)

3. Rotate logs
http://www.ginini.com.au/tools/rotatelogs/

Most of these tools are very configurable.

-HTH
Ramesh

[Bill McNAMARA_1]

There was an underlying reason that the log file filled up which was not critical but went unnoticed. This is a 24/7 system that has been running so far for a few months.

I have application software logs configurable to rotate rather than to fill applications and was hoping that syslog could do the same.

I didn't want to use cron because I just want peace of mind with this server and it is possible that in certain cases log files could rapidly increase..

I need a fixed config approach..

Later,
Bill

[Bill Hassell · Accepted solution]

Since syslog.log is plain ASCII, there's a lot you can do to manage it, except use syslogd. It has no options to manage it's logfiles (remember that this is *old* Unix code). Also note that syslogd is not a simple logging tool - syslog.conf can tell systlog to log all warnings and higher to 3 different files, 4 users, two serial ports and the console plust send the messages to another server to be included in another syslog file.

So the answer is: ceate a syslog scanner. Use this scanner to locate error messages of interest. Here's a start:

/usr/bin/grep -Fi -e error -e warn -e fatal -e fail /var/adm/syslog/syslog.log

Note the use of -F for much faster searching (I know grep -E can do the -e options but the performance penalty is enormous). Run this as often as you think necessary to notify sysadmins about issues. However, to avoid duplicate messages, you'll need to use a marker to start the search forward. See the man page for logger.

Then you can trim the syslog file. For history purposes, I keep the last 5 syslogs and roll them whenever syslog.log gets bigger than 2 megs. By running the error scanner independently of the log roller (pun intended), both tasks keep the logs under control. An example of a syslog scanner (and other log monitoring) is found in nightowl, written by David Totsch and located at:

ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/coolscripts/

You'll also find a bunch of other nifty tools in that directory. Feel free to browse.

[Paula J Frazer-Campbell]

Hi Bill

This works fine, I know you wre adverse to cron but cron it once a day:-


----------------cut here--------------
######################################
# Watcher for syslog
# PJFC 2001
######################################
# How big is it ?
a=`ls -l /var/adm/syslog/syslog.log | awk '{print $5}'`
if (($a > 100000))
then
mv /var/adm/syslog/syslog.log /var/adm/syslog/syslog.big
cat /dev/nul > /var/adm/syslog/syslog.log
mailx -s "Check the BIG syslog.big file" paula@avro.co.uk fi
----------------cut here--------------

The /tmp/team file contains one line which is our auto signature for the unix team.

"Unix Support Team"

Change the email address as I do not need to know if your syslog.log is big.

Also change the size to suit your requirements.


Paula

[Paula J Frazer-Campbell]

Hi Bill

Just a thought :-

If you have a login that you know is used every day then put a variation of the clean up script in their .profile - possibly root??

Paula

[Paula J Frazer-Campbell]

Bill

Inverse logic :-

Instead of syslog.log why not monitor the file systems.

i.e "avail" from a bdf output.

Safer and a trend can be established and monitored.

Paula

[Bill McNAMARA_1]

I tend to go with Bill's suggestions... the system wasn't really logged into for months, rather just providing a telecom platform service administered via web access.

The monitoring based on EMS brought my attention to the problem, but the system operators, normally don't want to know about things like this.. or even that the platform is based on unix for that matter. I prefer non intrusive reactive monitoring as opposed to scheduled monitoring as in most cases there can be many many systems to monitor and I don't like warnings developing at all!

Later,
Bill