Recommended syslog.conf settings when using opcle / OVO software? Too many events

[bstockli]

Our organization is using HP-UX / Solaris with OVO software and the java-based message browser.

Our syslog.conf on managed nodes is currently set to *.info;mail.none for opcle-monitored logs.

What we frequently see is several thousands (sometimes hundreds of thousands) of events for single system issues (a disk failure for example.)

This makes it difficult for our operations teams to manage the events coming in.

I've searched the net for recommended settings / best practices but haven't found anything so far.

Ultimately, we want to see errors or warnings, but we don't need 1000 events for a single issue.

Can anyone provide some insight?

Thanks in advance

[RJHall]

You could add an entry to syslog.conf that sends the appropriate level of syslogd messages to a named pipe, then use a filter script to screen the entries from the pipe that you want to capture. But you'd need to know in advance what messages you need to filter.

[Bill Hassell]

syslog.conf has no mechanism to filter messages, only the facility and severity level. You should restart syslogd with the -v option added (ie, syslogd -D -v) and now the facility and level will be added to every message, like this:

Aug 26 13:13:29 3D:atl1 tftpd[27922]: send_file: recv: Can't assign requested ...
Aug 26 13:23:22 6D:atl1 tftpd[27910]: Timeout (no requests in 10 minutes)
Aug 27 02:02:13 6B:atl1 LVM[22229]: lvlnboot -v

The 2 letter code (3D, 6D, 6B) identifies the facility (alphabetic) and severity level (number).

In the above listing,3D means DAEMON at ERR level, 6D is DAEMON at INFO level and 6B is USER at INFO level.

KERN    A    EMERG    0
USER    B    ALERT    1
MAIL    C    CRIT     2
DAEMON  D    ERR      3
AUTH    E    WARNING  4
SYSLOG  F    NOTICE   5
LPR     G    INFO     6
NEWS    H    DEBUG    7
UUCP    I
CRON    J
LOCAL0  Q
LOCAL1  R
...
LOCAL6  W
LOCAL7  X

Hopefully, the remote messages use one of the LOCAL facilities, for instance,  X for LOCAL7. In that case, you can redirect every message with facility LOCAL7 to a separate file or as suggested, a named pipe which is then read by a script that filters duplicate messages.

To filter LOCAL7 to a separate file, you remove the facility from syslog.log and write it to another file, like this:

*.info;mail.none;local0.none    /var/adm/syslog/syslog.log
mail.debug      /var/adm/syslog/mail.log
local7.info     /var/adm/syslog/ovo.log

VERY IMPORTANT - use NO spaces, only tabs. A space treats the entire line as a comment without notice.

Hint: cat -t /etc/syslog.conf

With syslogd -v turned on, you'll find a *lot* of messages that can be redirected to other files:

# local0 (krsd, ipmon) ipmon.log
# local1 (ems) to /var/adm/syslog/ems.log
# local2 (ignite) to /var/adm/syslog/ignite.log
# local5 (ftpd) to /var/adm/syslog/local5.log

Most of these facilities are hard to find in the docs...but syslogd -v will show them.

[BSTOCKLI1]

Excellent,  Thanks!

I tried a few things in our testbed environment and found the -v syslogd option results in less messages in syslog vs -a.

Ultimately I want to turn the facility/severity printing on everywhere so we can get a better idea of how to deal with the situation.

Here's a sample of one of the biggest event floods we've seen (easily over 1000 messages in one second:)

Jan 15 18:05:32 2A:hostname vmunix: class : lunpath, instance 6
Jan 15 18:05:32 2A:hostname vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Jan 15 18:05:32 2A:hostname vmunix:
Jan 15 18:05:32 2A:hostname vmunix: class : lunpath, instance 19
Jan 15 18:05:32 2A:hostname vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Jan 15 18:05:32 2A:hostname vmunix:
Jan 15 18:05:32 2A:hostname vmunix: class : lunpath, instance 13
Jan 15 18:05:32 2A:hostname vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Jan 15 18:05:32 2A:hostname vmunix:
Jan 15 18:05:32 2A:hostname vmunix: class : lunpath, instance 10
Jan 15 18:05:32 2A:hostname vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Jan 15 18:05:32 2A:hostname vmunix:

Once I make sure the -v syslogd changes don't break anything with existing opcle message generation, i'm going to try making separate syslog files as you mention and then use different opcle templates to filter these messages.