HP Labs Notification Mechanism Question

[Brad Klein]

There are many mechanisms available to software to notify a system administrator that something is wrong with a system. We're trying to understand from our existing customers, which of these mechanisms is most likely to be noticed and which of these mechanisms is preferred. We would greatly appreciate if you would take the time to rank these mechanisms from most likely to be noticed to least likely to be noticed. Also, if the most likely to be noticed is not the preferred mechanism, please tell us why and let us know which mechanism is preferred.

If someone else has already answered this question, we are still interested in your answer, even if it is the same; we would like to gather this information from a large number of administrators, so every bit helps.

Thanks!

Notification mechanisms:
------------------------------------
e-mail to root at localhost
wall message / message to console
message in syslog
e-mail to a user-supplied e-mail address
Openview/ITO event
SNMP trap
blinking lights
error during startup script execution
message in a product-specific logfile

[Chris Wilshaw · Accepted solution]

1 e-mail to a user-supplied e-mail address
2 message in syslog
3 message in a product-specific logfile
4 SNMP trap
5 wall message / message to console
6 e-mail to root at localhost
7 error during startup script execution
8 Openview/ITO event
9 blinking lights

[Paula J Frazer-Campbell]

Hi

With the advent of phones that hanle email my system email warning are sent to sysadmins and duty office.

I have set up only failures to mail out and servers monitor each other.

Logs, blinking lights etc are fine if you are either in the office or telnet in whereas a small script can monitor each log and warn of errors - plus the server checking itself and other servers is automatic and does not get forgotten.


Paula

[Victor_5]

Let me say ranking from 1 to 10, 10 means I use it very often.

1. e-mail to root at localhost 8

2. wall message / message to console 6

3. message in syslog 7

4. e-mail to a user-supplied e-mail address 9

5. Openview/ITO event 10

6. SNMP trap 6

7. blinking lights 7

8. error during startup script execution 8

9. message in a product-specific logfile 9

One more thing is EMS, I rank it as 10

Hope it is useful for you.

[Christopher McCray_1]

Hello,

1. Openview/ITO event -- NOT PREFFERRED because our Openview is not managed by us and the person who does is a trained(?) ape!!

2.e-mail to root at localhost -- root's email is checked often by us.

3. message in syslog -- same as above.

4. e-mail to a user-supplied e-mail address -- good as a secondary notification.

5. wall message / message to console and blinking lights -- Servers/consoles are on a different floor and would be hard to notice regularly.

6. error during startup script execution -- depends on person rebooting as to whether or not he/she will be at the console at startup.

7. message in a product-specific logfile -- potential problems with teaching certain people here about them and their locations.

SNMP trap -- N/A -- SNMP is turned off here.

Hope this helps and I'm availiable for further elaboration.

Chris

[James Beamish-White]

1 message in syslog
2 message in a product-specific logfile
3 SNMP trap
4 wall message / message to console
5 e-mail to a user-supplied e-mail address
6 e-mail to root at localhost
7 error during startup script execution
8 Openview/ITO event
9 blinking lights

I prefer this order because scripts and tools that monitor logs are easily available, while SNMP tools/Openview etc are more expensive and are hence less used.

Cheers!
James

[Jeff Schussele]

Brad,

Here you go - 1 being highest

1) e-mail to a user-supplied e-mail address
2) message in syslog
3) e-mail to root at localhost
4) wall message / message to console
5) Openview/ITO event
6) message in a product-specific logfile
7) error during startup script execution
8) blinking lights
9) SNMP trap

It's always a good thing when the end-user community is engaged in the planning stages.

Rgds,
Jeff

[Tom Maloy]

e-mail to root at localhost (checked frequently)
e-mail to a user-supplied e-mail address (can be used to notify group of admins)
message in syslog (checked frequently)
error during startup script execution
message in a product-specific logfile
Openview/ITO event
SNMP trap
blinking lights
wall message / message to console

My $0.02...

Tom

[A. Clay Stephenson]

I'll rank these in the order I woulf notice them:
1)== most;


1) Openview/ITO event - except the politically correct term is now VP/O

2) SNMP trap

I assume both 1) & 2) are logged by VP/O and if severe enough and not acknowledged an automatic noticifaction is sent via pager.

3) e-mail to a user-supplied e-mail address
4) e-mail to root at localhost
5) wall message / message to console
6) message in syslog - this would probably be caught by VP/O anyway
7) message in a product-specific logfile - again probably caught by VP/O
8) error during startup script execution
9) blinking lights

[S.K. Chan]

1 being the most prefered method or most frequently used. If not in the list it means we do not use it at all.
1) e-mail to a user-supplied e-mail address and paging notification
2) e-mail to root at localhost (root on all local host has its aliases) and paging notification.
3) dmesg
4) message in syslog
5) error during startup script execution
6) message in a product-specific logfile
7) wall message/message to console
8) blinking lights

[John Payne_2]

Preferred:

**email to a user-supplied e-mail address.** Why you say? I have way too many servers to be looking through everything on every box...

Other acceptable ways, but less preferred: (In order of there rank, most to least preferred.)

1) message to syslog
2) SNMP trap
3) blinking lights - Would have to wait until someone noticed. Could be days or weeks.

Not preferred at all:

wall message/message to console - We are a lights-out shop. The console message would not be seen, the walled message would panic end users.

Email to root at localhost. -
I would rather have it come direct to me. (Although the feature of emailing me and root would be nice in the event that something happens to me, and my boss desides to try to figure out what is wrong...)

Openview/ITO - We don't currently run openview. If we did, I would be totally for it.

error during startup script execution - For the same reason as console message, also, I would have to remember to go to /etc/rc.log all the time looking for messages.

message in a product-specific logfile - Isn't this the same as sending to the syslog? This just adds unneeded complexity in my opinion.

Hope it helps.

John

[H.Merijn Brand (procura]

1. e-mail to root
+ message on console + message on GSP
2. message on console
+ message in syslog
3. message in product specific logfile
+ in serious cases: on each startup of the product

IMHO the rest is not interesting enough

my ??? 0.05

[Christopher Caldwell]

4 e-mail to root at localhost

9 wall message / message to console

2 message in syslog

6 e-mail to a user-supplied e-mail address

5 Openview/ITO event

1 SNMP trap

8 blinking lights

7 error during startup script
execution

3 message in a product-specific logfile

[Patrick Wallek]

My ranking from first to last:

e-mail to a user-supplied e-mail address
e-mail to root at localhost
message in syslog
message in a product-specific logfile
error during startup script execution
Openview/ITO event
SNMP trap
blinking lights
wall message / message to console
I would probably rank the Openview and SNMP stuff higher if we had openview running at the last job, which we didn't.

The blinking lights and wall message / console message to me would be pretty useless.

[Daimian Woznick]

The following is 1 being the highest and most preferred:

1. e-mail to user-supplied e-mail address (e-mail to pager preferred)
2. Openview/ITO event
3. Message in syslog
4. Message in product-specific logfile

And the rest can just be placed at the end in a group.

[Deshpande Prashant]

Hi
I, too, agree with majority,
email /page to user-supplied address is most preferred method in our env.
Rest as follows.

2) Message in syslog.
3. Message in product specific log file.
4) e-mail to root at localhost (root messages from all machines get routed to single machine for viewing.)
5) dmesg
6) error during startup script execution
7) SNMP Trap
8) blinking lights
9) wall message/message to console (Web console in use and rarely viewed)


Thanks.
Prashant Deshpande.

[Martin Johnson]

Openview/ITO event.

We are trying to get to using a single screen to determine the health of our environment. Ultimately, it will be service navigator, part of openview. We will be using Telalert for paging from ITO.

Performance and availability will monitored. MeasureWare, NetIQ and Sitescope are used to monitor performance. We are looking into precise's suite of tools for possible inclusion of performance monitoring.

The syslog is monitored by ITO (logfile encapsulation) and we do have scripts that write to the syslog.

We are using SPI plugins to monitor SAP, DBs and weblogic.

In the near future, forward NNM messages to ITO. Use ECS to manage the volume of messages.

The ultimate goal is for operations to handle level I and level II suport, with SAs doing level III. We are probably 1-2 years away from


HTH
Marty

[George A Bodnar]

1) Openview/ITO event
** This gets logged for serious events

2) e-mail to user
** Will get quick response, but needs maintained as people switch jobs/roles, etc.

3) syslog message
** This becomes ITO message (or equivalent tool) in most environments and is monitored

4) root email
** Not checked as often - actually we route to ITO filters

5) SNMP trap
** Monitor tool might catch

6) error during startup script execution
** Only found if watching boot or auditing the rc log

7) Blinking lights
** Not watched all the time - very likely to sit for quite a while

8) wall message
** Not likely to be seen

[Tom Danzig]

1 - Hot operator on head with hammer (J/K)
-------------------------
1) e-mail to a user-supplied e-mail address
2) message in a product-specific logfile
3) message in syslog
4) Openview/ITO event
5) error during startup script execution
6) SNMP trap
7) e-mail to root at localhost
8) wall message / message to console
9) blinking lights

[Bill McNAMARA_1]

message in a product-specific logfile
message in syslog
error during startup script execution
SNMP trap
Openview/ITO event
blinking lights
wall message / message to console
e-mail to a user-supplied e-mail address
e-mail to root at localhost


Of course 1 is not your responsibility, although interface to nettl should be easier and better documented.. my apps log to nettl, and nettl often gets corrupt on multiple fast logging, log files overflow and fill up filesystems, I'd really appreciate a log rotation of syslog in particular..


from a sysadmin point of view most errors will be id'd from application point of view, symptomed in syslog and/or snmp, identified via application logs and if in the case of HW/panic etc.. id'd at the system from led status.

I really don't like the email to root, in fact I link it to /dev/null... so I don't know what I'm missing but at least my mailbox doesn't fill up like crazy on error!


Later,
Bill

[Hai Nguyen_1]

Brad,
Here is my ranking:


10. wall message / message to console
9. message in syslog
8. error during startup script execution
7. message in a product-specific logfile
6. e-mail to root at localhost
5. e-mail to a user-supplied e-mail address
4. blinking lights
3. penview/ITO event
2. SNMP trap

Hai

[Ian Kidd_1]

Since I work in an environment where there are several admins for multiple boxes (including some servers managed remotely), messages to root@localhost, as well as blinking lights and console messages are of least importance. Here's my preference:

(1) e-mail to a user-supplied e-mail address
(2) message in syslog
(3) message in a product-specific logfile
(4) error during startup script execution
(5) SNMP trap
(6) Openview/ITO event
(7) e-mail to root at localhost
(8) blinking lights
(9) wall message / message to console

[Michael Tully]

Current Rank from highest to lowest

e-mail to a user-supplied e-mail address
message in syslog
EMS logs
error during startup script execution
blinking lights/console messages

Our environment is a rapidly changing one where we are implementing a whole lot of new gear, unfortunately some faster than others.
We would like to be using SNMP traps to operators in the near future if the project ever gets moving. Our consoles are ioslated in data centre computer rooms, not all accessible by the people who mater... (us admins)
We would not like wall messages as we use terminals in our many warehouses which would scare the crap out of a forklift driver....

[Wodisch_1]

Hi,

my rating goes from 0=top to 9=bottom priority:

0) soundfile played after call ing the mobile phone (or pager)
1) e-mail to a user-supplied e-mail address
2) Openview/OVO event
3) blinking lights
4) SNMP trap
5) message in syslog
6) wall message / message to console
7) e-mail to root at localhost
8) message in a product-specific logfile
9) error during startup script execution

HTH,
Wodisch

[Steven Sim Kok Leong]

Hi,

In order of top priority:

1) message in syslog (because I wrote an interface to page on syslog logger trigger)
2) e-mail to a user-supplied e-mail address
3) wall message / message to console
4) message in a product-specific logfile
5) error during startup script execution
6) blinking lights
7) e-mail to root at localhost
8) SNMP trap
9) Openview/ITO event

Hope this helps. Regards.

Steven Sim Kok Leong