SCSI disk error should send an e-mail to root

[Geert Van Pamel]

My system generated a nonrecoverable SCSI read error last night.

I would have expected that root should have received an error mail from the event logging subsystem?

The SCSI read error has been only logged with priority 400. The standard /etc/evmlogger.conf only sends an error mail to root if the priority is at least 600.

I would have expected that a non recoverable SCSI read error would have been logged with at least priority 600.

Is there any possibility to change the priority of event sys.unix.binlog.hw.scsi?


evmget -f "[ since 2009:09:20:03:00:00 ] & [ before 2009:09:20:04:00:00 ]" |evmsort |evmshow -d |pg

======================= Binary Error Log event =======================
EVM event name: sys.unix.binlog.hw.scsi

Binary error log events are posted through the binlogd daemon, and
stored in the binary error log file, /var/adm/binary.errlog. This
event is used to report all SCSI device errors, including disk,
tape, HSZ raid events and adapter errors.

Action: Use Compaq Analyze or DECevent to read and analyze the
system error log to determine if a SCSI device may need to be
replaced.

Formatted Message:
SCSI event

Event Data Items:
Event Name : sys.unix.binlog.hw.scsi
Priority : 400
Timestamp : 20-Sep-2009 03:42:55
Host Name : hbsitcca
Format : SCSI event
Reference : cat:evmexp.cat:300

Variable Items:
subid_class (INT32) = 199
subid_num (INT32) = 0
subid_unit_num (INT32) = 24
subid_type (INT32) = 0
binlog_event (OPAQUE) = [OPAQUE VALUE: 1144 bytes]

============================ Translation =============================
Sequence number of error: -499735687
Time of error entry: 20-Sep-2009 03:42:55
Host name: hbsitcca

SCSI CAM ERROR PACKET
SCSI device class: DISK
Bus Number: 0
Target number: 3
Lun Number: 0

Name of routine that logged the event: cdisk_check_sense
Event information: Medium Error at block number: 44165980
Hardware detected event: Hard Error Detected
Event information: Hardware ID = 74
Device Name: COMPAQ BD036659CC 3B00
Event information: Active CCB at time of error
Event information: CCB request completed with an error

############### Entry End ###############

Event information: Error, exception, or abnormal condition
Event information: MEDIUM ERROR - Nonrecoverable medium error

############### Entry End ###############

[Geert Van Pamel]

Sorry, there is still an open question.

How could I change the EVM configuration and set the default priority of event sys.unix.binlog.hw.scsi to 600 instead of 400 (default value)?

I want to receive an e-mail to the root account anytime a SCSI disk error happens.

Note that SCSI I/O errors are currently logged with priority 400. The standard /etc/evmlogger.conf only sends an error mail to root if the priority is at least 600.

Thanks to look into this?

I have been reading the Tru64 UNIX Troubleshooting book by Martin Moore & Steven Hancock, but I do not find a solution (ISBN 1-55558-274-5 from Digital Press).

[cnb]

I know nothing about evm, but I happened to run across this looking for another issue. So I may be way off base here, so apologies in advance if this is not what you're looking for.

Why not change the filter to mail events with a lower priority in evmlogger.conf?


filter "[prio >= 600] & ![name @SYS_VP@.syslog.mail]"

or maybe write a secondary evmlogger.conf file with specific filtering?

http://www.tru64unix.compaq.com/docs/best_practices/EVM_BP/TITLE.HTM

from the evmlogger.conf man page:

This example initiates an instance of the evmlogger command with the fol-
lowing configuration:

1. Binary events are written to a file in the /var/evm/evmlog directory
named evmlog.xxx where xxx is the current year, month, and day -- for
example, /var/evm/evmlog/evmlog.19981217. An alternate log path is
specified in case of write failures to the primary path.

2. A new generation of the log is started automatically if the size
exceeds 256 Kbytes.

3. All events with a priority of at least 200 are selected for logging.

4. Duplicate events are suppressed.

5. Events with a priority of at least 600 are displayed on the system
console as formatted events, showing the timestamp, the priority and
the event's message.

6. Events with a priority of at least 600 are also mailed to root.

7. A maximum of 20 events will be queued for forwarding to root when an
instance of the forwarding command is already running.

eventlog
{
name evmlog
logfile /var/evm/evmlog/evmlog.dated
type binary
maxsize 256 # Kbytes

alternate /altlogs/evmlog/evmlog.dated

# Log all events with priority >= 200:
filter "[prio >= 200]"

# Suppress logging of duplicate events:
suppress
{ filter "[name *]"
period 30 # minutes
threshold 3 # No. of instances before suppression
}

}

# Log high-priority events to the system console:
eventlog {
name console_log
logfile /dev/console
filter "[prio >= 600]"
type formatted
show_template "@timestamp [@priority] @@"
}

# Forward details of high-priority events to root:
forward {
name priority_alert
# Don't forward mail events through mail (see note above):
filter "[prio >= 600] & ![name @SYS_VP@.syslog.mail]"

suppress
{ filter "[name *]"
period 120 # minutes
threshold 1 # No. of duplicates before suppression
}

# This evmshow command writes a subject line as the first line of
# output, followed by a detailed display of the contents of the
# event. The resulting message is distributed by mail(1).

command "evmshow -d -t 'Subject: EVM ALERT [@priority]: @@' \
| mail root"
}

FILES

/etc/evmlogger.conf
Location of the EVM logger configuration file.

/var/evm/adm/config/logger
Default location of the secondary EVM logger configuration files.


HTH

Rgds,

[cnb · Accepted solution]

I found another example file:

http://www.tru64unix.compaq.com/docs/best_practices/BP_EVM_CELL/TITLE.HTM

Rgds,

[Geert Van Pamel]

I did not change the standard /etc/evmlogger.conf file.

Rather I added another "local" independent conf file.

cd /var/evm/adm/config/logger

vi sysman.conf

# Forward details of high-priority events to root:
forward {
name sysman_forward

# Don't forward mail events through mail (see note above).
filter "[prio >= 400] & ![name @SYS_VP@.syslog.mail]"

suppress
{ filter "[name *]"
period 120 # minutes
threshold 1 # No. of duplicates before suppression
}

# This evmshow command writes a subject line as the first line of
# output, followed by a detailed display of the contents of the event.
# The resulting message is distributed by mail(1).
command "evmshow -d -t 'Subject: @host_name [@priority] @timestamp @@' | mail sysman"

# Limit the number of events that can be queued for this command:
maxqueue 100
}

evmreload -l

evmpost -p 400 -a "test message to mobile"

It forwards an alarm message via e-mail to an alias sysman. In my case, this sends an SMS message to my mobile.

The command "mail sysman" could be replaced by any command forwarding an alert message using any possible network protocol.

TCP is a wonderful swiss knife tool! Still alive and kicking after 35 years!

Thanks to all of you!