EMS events run a script

[Rick Garland]

Hi all:

I am looking to have EMS alerts run a script depending on the severity level. How can I?

Thanks

[Patrick Wallek]

What about sending an e-mail, like your previous post, and for that ID have a .forward file that then executes a script.

[Steve Post]

one way:
cd /etc/opt/resmon/lbin

run ./monconfig
With this you can supposedly tell it to run a script. But that's not what I did. I told it to send to an email.

The email address I made is a mail alias to pipe to a script I called "emsfilter".

When an event happens, an email goes to user emsalert. Ems alert is an alias in file /etc/mail/aliases. It pipes the email to /usr/local/bin/emsfilter.

Script emsfilter parses through the email, pulls out the "Event time, Severity, Monitor, and system." It sends this much smaller text out to a set of pagers and cell phones (all with email addresses).

To grab the email I ran this in the script.
cat - > /tmp/ems_mail.${$}

Now /tmp/ems_mail. holds my text to parse through.

Before I did this I would get 10 lame pages passed to me at random times. I would have to write down the snippets of text and try to get a message out of it. It took too long to get legitimate information. Now the message fits into ONE page.

So in summary,
-- run "monconfig" to tell it to send events to a email address like emsalert.
-- Edit file /etc/mail/aliases to have "emsalert" pipe to file /usr/local/emsfilter.
-- create some script like emsfilter to parse through an email and send just the juicy bits out to the pagers.

Let me know if you want more info.

steve

[Rick Garland]

Thanks for the responses.

Question 1, if I am working from the EMS GUI and want to set filesystem thresholds, how can I tell it to email based on the severity?

Question 2, the monconfig appears to be for HW monitoring, what if I want to monitor /system/filesystem/availMb? I do not see that option.

[Steve Post]

1. I never knew about a GUI version of EMS. I am using hpux11.0. Maybe you have a better version.
2. It looks like you want to track disk usage and give you a warning if a file system goes over a certain threshold. My predecessor had a perl script for that. I tweeked it. I run the bugger every 15 minutes. I have a threshold file to I can it that 90% is ok on THIS file system but 86% is an alarm on THAT filesystem.
It uses the bdf command and (of course) email for paging out alarms.
But the nicest thing about the perl script is it's SIMPLE.
I'll send the bdf.pl file if you want it.

steve

[Rick Garland]

Hi Steve:

The /system/filesystem/availMb was an example. The EMS GUI has been around for some time. If you invoke SAM, there is an icon for Resource Management and in this section is the Event Monitoring Service. As can be seen, there are more options to monitoring a system that are not available via the monconfig option.

What I am looking to do is have a configuration that is going to give us the alerts based on severity but I am also looking to have lower level admins understand what is the process without going through so many hoops.

In other words, I will configure, document, and get working but I will not be maintaining this configuration forever.

(Gotta have something for the mid- and jr-level admins to do and learn.)

[Andrew Merritt_2]

Hi Rick,
I don't think the events sent by the Resource Monitoring EMS service have an associated severity, they just indicate that a condition has been met. The events sent by the EMS HW monitors do, however, but that's not what you're asking about.

For the resource monitoring events, such as AvailMB, you could have different email addresses that correspond to the levels of severity you want to use, and then do something appropriate with a script, as suggested by Steve above.

Andrew

[Rick Garland]

I was thinking that but I am hoping to find something a little cleaner to work with.

My goal, and I'm sure everyone else's goal, is to make things somewhat clean and simple so those coming up behind me have some understanding as to what is happening.

If all else fails then I will implement this option. I am hoping to find something else.

Forgive me, but I will keep trying.

Thanks!

[Andrew Merritt_2]

Rick,
I'm not sure what you're saying.

The events don't have a severity, so you obviously can't distinguish between them based on that.

Given that your original question doesn't make sense, what is the problem that you're trying to solve?

Andrwe

[Steve Post]

I know that working with the configuration via monconfig, you are in fact modifying a set of TEXT files at /var/stm/config/tools/monitor. I suspect that is the case for the gui too.

As far as how you feel about the different serverity levels for different components on the system? How about the meaning behind the metric? Or how the metric should be resolved? Or who is responsible to monitor each metric? That's up to you. For me, it's simple: Bad stuff = Page me.

At least here is that script.
And the line in /etc/mail/aliases is
"emspager : |/usr/local/bin/emsfilter"

And to activate this email alias you run "newaliases".

[Rick Garland]

To begin, there are 2 areas of EMS monitoring. There is HW monitoring via the /etc/opt/resmon/lbin/monconfig and there is OS monitoring via SAM,
Resource Monitoring-->Event Monitoring Service.

Example:

Got a filesystem that fills up, example, the arch logs directory for oracle. If this issue is not resolved in a timely manner then oracle will stop.

Within EMS I can set thresholds within the resource /system/filesystem/availMb and have an alert sent out.

There are no severity settings for EMS alerts except for the SNMP trap notification option.

I do not have NNM, ITO, etc. so the SNMP trap doesn't really help much.

So, I get a filesystem that reaches a threshold. Send me an alert depending on the severity which is directly related to the amount the filesystem has filled.

[Andrew Merritt_2]

Hi Rick,
As you've said, the Event notifications from the Resource Management do not have an associated Severity level, unless you use SNMP to deliver them.

If you want to have multiple levels of notification for one resource (which I think is not what was intended in the design of the Resource Management), then the only way I can see of doing this (without using SNMP) is to use email for notification, and have different email addresses for the different levels of severity you want. I don't see this as being a particularly complicated setup.

You're the one who has decided to have different severity levels for different thresholds. You might want to review this; what do you want to have happen when the first one is reached? E.g. if you decide that 80% full is your first event; do you want someone to free space up? If not, what is the purpose of the notification? If so, then space will get freed, and you won't need any further notifications.

[Rick Garland]

Many thanks all!

Have so many contraints in the configuration of system monitoring. The biggest constraint is not having any type of working monitoring application. I am going to look at nagios and big brother to see if I can get some more success there.

Again, many thanks to all!