topic Re: HP Labs Notification Mechanism Question in Operating System - HP-UX

HP Labs Notification Mechanism Question

Brad Klein — Fri, 26 Jul 2002 13:42:44 GMT

There are many mechanisms available to software to notify a system administrator that something is wrong with a system. We're trying to understand from our existing customers, which of these mechanisms is most likely to be noticed and which of these mechanisms is preferred. We would greatly appreciate if you would take the time to rank these mechanisms from most likely to be noticed to least likely to be noticed. Also, if the most likely to be noticed is not the preferred mechanism, please tell us why and let us know which mechanism is preferred.

If someone else has already answered this question, we are still interested in your answer, even if it is the same; we would like to gather this information from a large number of administrators, so every bit helps.

Thanks!

Notification mechanisms:
------------------------------------
e-mail to root at localhost
wall message / message to console
message in syslog
e-mail to a user-supplied e-mail address
Openview/ITO event
SNMP trap
blinking lights
error during startup script execution
message in a product-specific logfile

Re: HP Labs Notification Mechanism Question

Chris Wilshaw — Fri, 26 Jul 2002 13:46:36 GMT

1 e-mail to a user-supplied e-mail address
2 message in syslog
3 message in a product-specific logfile
4 SNMP trap
5 wall message / message to console
6 e-mail to root at localhost
7 error during startup script execution
8 Openview/ITO event
9 blinking lights

Re: HP Labs Notification Mechanism Question

Paula J Frazer-Campbell — Fri, 26 Jul 2002 13:50:23 GMT

Hi

With the advent of phones that hanle email my system email warning are sent to sysadmins and duty office.

I have set up only failures to mail out and servers monitor each other.

Logs, blinking lights etc are fine if you are either in the office or telnet in whereas a small script can monitor each log and warn of errors - plus the server checking itself and other servers is automatic and does not get forgotten.

Paula

Re: HP Labs Notification Mechanism Question

Victor_5 — Fri, 26 Jul 2002 13:51:51 GMT

Let me say ranking from 1 to 10, 10 means I use it very often.

1. e-mail to root at localhost 8

2. wall message / message to console 6

3. message in syslog 7

4. e-mail to a user-supplied e-mail address 9

5. Openview/ITO event 10

6. SNMP trap 6

7. blinking lights 7

8. error during startup script execution 8

9. message in a product-specific logfile 9

One more thing is EMS, I rank it as 10

Hope it is useful for you.

Re: HP Labs Notification Mechanism Question

Christopher McCray_1 — Fri, 26 Jul 2002 13:56:26 GMT

Hello,

1. Openview/ITO event -- NOT PREFFERRED because our Openview is not managed by us and the person who does is a trained(?) ape!!

2.e-mail to root at localhost -- root's email is checked often by us.

3. message in syslog -- same as above.

4. e-mail to a user-supplied e-mail address -- good as a secondary notification.

5. wall message / message to console and blinking lights -- Servers/consoles are on a different floor and would be hard to notice regularly.

6. error during startup script execution -- depends on person rebooting as to whether or not he/she will be at the console at startup.

7. message in a product-specific logfile -- potential problems with teaching certain people here about them and their locations.

SNMP trap -- N/A -- SNMP is turned off here.

Hope this helps and I'm availiable for further elaboration.

Chris

Re: HP Labs Notification Mechanism Question

James Beamish-White — Fri, 26 Jul 2002 14:12:02 GMT

1 message in syslog
2 message in a product-specific logfile
3 SNMP trap
4 wall message / message to console
5 e-mail to a user-supplied e-mail address
6 e-mail to root at localhost
7 error during startup script execution
8 Openview/ITO event
9 blinking lights

I prefer this order because scripts and tools that monitor logs are easily available, while SNMP tools/Openview etc are more expensive and are hence less used.

Cheers!
James

Re: HP Labs Notification Mechanism Question

Jeff Schussele — Fri, 26 Jul 2002 14:16:18 GMT

Brad,

Here you go - 1 being highest

1) e-mail to a user-supplied e-mail address
2) message in syslog
3) e-mail to root at localhost
4) wall message / message to console
5) Openview/ITO event
6) message in a product-specific logfile
7) error during startup script execution
8) blinking lights
9) SNMP trap

It's always a good thing when the end-user community is engaged in the planning stages.

Rgds,
Jeff

Re: HP Labs Notification Mechanism Question

Tom Maloy — Fri, 26 Jul 2002 14:18:01 GMT

e-mail to root at localhost (checked frequently)
e-mail to a user-supplied e-mail address (can be used to notify group of admins)
message in syslog (checked frequently)
error during startup script execution
message in a product-specific logfile
Openview/ITO event
SNMP trap
blinking lights
wall message / message to console

My $0.02...

Tom

Re: HP Labs Notification Mechanism Question

A. Clay Stephenson — Fri, 26 Jul 2002 14:22:26 GMT

I'll rank these in the order I woulf notice them:
1)== most;

1) Openview/ITO event - except the politically correct term is now VP/O

2) SNMP trap

I assume both 1) & 2) are logged by VP/O and if severe enough and not acknowledged an automatic noticifaction is sent via pager.

3) e-mail to a user-supplied e-mail address
4) e-mail to root at localhost
5) wall message / message to console
6) message in syslog - this would probably be caught by VP/O anyway
7) message in a product-specific logfile - again probably caught by VP/O
8) error during startup script execution
9) blinking lights

Re: HP Labs Notification Mechanism Question

S.K. Chan — Fri, 26 Jul 2002 14:25:51 GMT

1 being the most prefered method or most frequently used. If not in the list it means we do not use it at all.
1) e-mail to a user-supplied e-mail address and paging notification
2) e-mail to root at localhost (root on all local host has its aliases) and paging notification.
3) dmesg
4) message in syslog
5) error during startup script execution
6) message in a product-specific logfile
7) wall message/message to console
8) blinking lights

Re: HP Labs Notification Mechanism Question

John Payne_2 — Fri, 26 Jul 2002 14:46:12 GMT

Preferred:

**email to a user-supplied e-mail address.** Why you say? I have way too many servers to be looking through everything on every box...

Other acceptable ways, but less preferred: (In order of there rank, most to least preferred.)

1) message to syslog
2) SNMP trap
3) blinking lights - Would have to wait until someone noticed. Could be days or weeks.

Not preferred at all:

wall message/message to console - We are a lights-out shop. The console message would not be seen, the walled message would panic end users.

Email to root at localhost. -
I would rather have it come direct to me. (Although the feature of emailing me and root would be nice in the event that something happens to me, and my boss desides to try to figure out what is wrong...)

Openview/ITO - We don't currently run openview. If we did, I would be totally for it.

error during startup script execution - For the same reason as console message, also, I would have to remember to go to /etc/rc.log all the time looking for messages.

message in a product-specific logfile - Isn't this the same as sending to the syslog? This just adds unneeded complexity in my opinion.

Hope it helps.

John

Re: HP Labs Notification Mechanism Question

H.Merijn Brand (procura — Fri, 26 Jul 2002 15:11:02 GMT

1. e-mail to root
+ message on console + message on GSP
2. message on console
+ message in syslog
3. message in product specific logfile
+ in serious cases: on each startup of the product

IMHO the rest is not interesting enough

my ??? 0.05

Re: HP Labs Notification Mechanism Question

Christopher Caldwell — Fri, 26 Jul 2002 16:35:14 GMT

4 e-mail to root at localhost

9 wall message / message to console

2 message in syslog

6 e-mail to a user-supplied e-mail address

5 Openview/ITO event

1 SNMP trap

8 blinking lights

7 error during startup script
execution

3 message in a product-specific logfile

Re: HP Labs Notification Mechanism Question

Patrick Wallek — Fri, 26 Jul 2002 16:43:28 GMT

My ranking from first to last:

e-mail to a user-supplied e-mail address
e-mail to root at localhost
message in syslog
message in a product-specific logfile
error during startup script execution
Openview/ITO event
SNMP trap
blinking lights
wall message / message to console
I would probably rank the Openview and SNMP stuff higher if we had openview running at the last job, which we didn't.

The blinking lights and wall message / console message to me would be pretty useless.

Re: HP Labs Notification Mechanism Question

Daimian Woznick — Fri, 26 Jul 2002 16:58:30 GMT

The following is 1 being the highest and most preferred:

1. e-mail to user-supplied e-mail address (e-mail to pager preferred)
2. Openview/ITO event
3. Message in syslog
4. Message in product-specific logfile

And the rest can just be placed at the end in a group.

Re: HP Labs Notification Mechanism Question

Deshpande Prashant — Fri, 26 Jul 2002 17:28:21 GMT

Hi
I, too, agree with majority,
email /page to user-supplied address is most preferred method in our env.
Rest as follows.

2) Message in syslog.
3. Message in product specific log file.
4) e-mail to root at localhost (root messages from all machines get routed to single machine for viewing.)
5) dmesg
6) error during startup script execution
7) SNMP Trap
8) blinking lights
9) wall message/message to console (Web console in use and rarely viewed)

Thanks.
Prashant Deshpande.

Re: HP Labs Notification Mechanism Question

Martin Johnson — Fri, 26 Jul 2002 18:52:28 GMT

Openview/ITO event.

We are trying to get to using a single screen to determine the health of our environment. Ultimately, it will be service navigator, part of openview. We will be using Telalert for paging from ITO.

Performance and availability will monitored. MeasureWare, NetIQ and Sitescope are used to monitor performance. We are looking into precise's suite of tools for possible inclusion of performance monitoring.

The syslog is monitored by ITO (logfile encapsulation) and we do have scripts that write to the syslog.

We are using SPI plugins to monitor SAP, DBs and weblogic.

In the near future, forward NNM messages to ITO. Use ECS to manage the volume of messages.

The ultimate goal is for operations to handle level I and level II suport, with SAs doing level III. We are probably 1-2 years away from

HTH
Marty

Re: HP Labs Notification Mechanism Question

George A Bodnar — Fri, 26 Jul 2002 18:58:28 GMT

1) Openview/ITO event
** This gets logged for serious events

2) e-mail to user
** Will get quick response, but needs maintained as people switch jobs/roles, etc.

3) syslog message
** This becomes ITO message (or equivalent tool) in most environments and is monitored

4) root email
** Not checked as often - actually we route to ITO filters

5) SNMP trap
** Monitor tool might catch

6) error during startup script execution
** Only found if watching boot or auditing the rc log

7) Blinking lights
** Not watched all the time - very likely to sit for quite a while

8) wall message
** Not likely to be seen

Re: HP Labs Notification Mechanism Question

Tom Danzig — Fri, 26 Jul 2002 19:31:08 GMT

1 - Hot operator on head with hammer (J/K)
-------------------------
1) e-mail to a user-supplied e-mail address
2) message in a product-specific logfile
3) message in syslog
4) Openview/ITO event
5) error during startup script execution
6) SNMP trap
7) e-mail to root at localhost
8) wall message / message to console
9) blinking lights

Re: HP Labs Notification Mechanism Question

Bill McNAMARA_1 — Fri, 26 Jul 2002 19:41:37 GMT

message in a product-specific logfile
message in syslog
error during startup script execution
SNMP trap
Openview/ITO event
blinking lights
wall message / message to console
e-mail to a user-supplied e-mail address
e-mail to root at localhost

Of course 1 is not your responsibility, although interface to nettl should be easier and better documented.. my apps log to nettl, and nettl often gets corrupt on multiple fast logging, log files overflow and fill up filesystems, I'd really appreciate a log rotation of syslog in particular..

from a sysadmin point of view most errors will be id'd from application point of view, symptomed in syslog and/or snmp, identified via application logs and if in the case of HW/panic etc.. id'd at the system from led status.

I really don't like the email to root, in fact I link it to /dev/null... so I don't know what I'm missing but at least my mailbox doesn't fill up like crazy on error!

Later,
Bill