Disk threshold notifications and generating reports problems

[deceit]

Hi,
I am configuring Disk thresholds across the servers and running into problems.
Running HP SIM 4.2 SP2. The Servers have Support Pack 7.40.

I have set the following:
---------
Critical disk usage threshold (percent): 90
Reset critical disk usage threshold at (percent): 80
Warning disk usage threshold (percent): 0
Reset warning disk usage threshold at (percent): 0
Agent polling interval (seconds): 120
---------
Replicating these settings to the machines seems to work correctly.
I have verified this by checking the File System Space used page on each server.
These graphs useem to update relatively quickly when i try and fill the disk up.
So here are the problems.

1. SNMP traps are not being sent back to the Management box. The extent of my SNMP testing has been running a tool from my local PC to both the management box and a test box to see if it connects and can read some values with the community strings I have set. This works OK.
I can see a syslog event being generated on the test machine when the threshold passes "90"

2. An Event is not generated when the threshold drops below "80"

3. Running a report on disksize/used/percent etc for all servers doesnt update in real time. I cannot work out what causes this data to update.

4. No email is sent. Do I have to configure an Automatic Event handling task for this?



Any help will be much appreciated.

Thanks

[Rob Buxton]

You don't mention OS so I'll assume Windows.
There are a couple of different mechanisms at work here.
When an Client Server Agents threshold is breached it generates an SNMP Trap that is sent to HPSIM. To test that SNMP Traps are getting to your HPSIM Server run up the HP Management Agent GUI from the Control Panel. There's an SNMP Send utility. That should generate a Major Event for the Server.
SNMP Trap destinations are configured in the SNMP Service.
Once an Event is generated then an Automatic task needs to be configured to react to those Events. You can start of with a generic task for all Critical Events and another for All Major Events.
As for the reporting data, that is not updated dynamically. Different processes update different tables. E.g. Many of the reporting tables are updated by the Bi-Weekly task. Others by the Hardware Polling task.
Check out the Scheduled Tasks.

[deceit]

Hi Rob,

Thanks for your reply.

Yes I am using Windows 2003 STD for both HPSIM and Clients. The hardware platform are DL320/360s G1-G4s.


I have tested sending several traps via the method you mentioned and this does not work.
I dont even see an event in the Uncleared Event Status sections.

I have configured Informational/Critical/Major notifications to take place.
I have tested the Informational notification and this emails fine.

I am not too familar with SNMP. Could there be an issue with the "SNMP Trap Service" that receives the traps? Is it possible to check that SNMP traps are reaching this, but not getting to HPSIM?

Thanks

[tmagee]

This sounds very like the issue I'd got with disk thresholds (see topic "disk threshold falling alert not generated, not reset", in Feb 05). I'd heard there was a fix due in agents 7.4, but as you're running this then looks like it hasn't been. (I've not tried it myself yet -I've now moved to sim 5.0 and agents 7.3).

I would get the alert when disk space increased but not when space was then reduced below the threshold level and then never afterwards if space increased - unless a reboot of the server was done and the same symptom would repeat.

It's as if the agent isn't reading the threshold value - this isn't visible either on the disk graphical interface (the red and yellow triangles) so that's always confused me.

[deceit]

I rebuilt and upgrade HPSIM to 5.0 on a free machine. The snmp issue above has been fixed but the issue with disk thresholds still remain.

Event viewer reports threshold exceeded.
No notification to the HPSIM Console saying it has been exceeded.
No event viewer/notification to say its resumed to a reasonable level.

I can live with not getting the second message, but it would be good for the HPSIM console to know when a machine has run out of space..


Any ideas guys?

[tmagee]

Are you now getting the snmp message resulting from the "send test trap" option in management agents icon in the control panel, in SIM event viewer and your email (as described by Rob Buxton)? If so this shows the community name and ip on both the SIM and the server being monitored are set up ok.

Is it still only the disk threshold alert that's not reaching SIM? (e.g. does a reboot of the server being monitored (if possible) generate other alerts, or some sample test, e.g. removing a network cable generate an alert?) And do email alerts reach you ok for these? If this is all as expected, in the management agents icon (in control panel), there's a services tab showing active agents - does this include the entry for "threshold support"?. There's a clear threshold tab there too, does that offer any differences?

And within SIM, under Options, Events, SNMP Trap Settings, check for Mib entry "Cpqthrsh.mib" - is this set for trap handling?

I've not tried agents 7.4 yet, but if all the above is looking ok it may still be a symptom of the threshold issue I was trying to get resolved.

[deceit]

tmagee,

I have done some more testing and the SNMP functionality seems to be not working again.

I have run a third party SNMP daemon on my local pc as well as the server and can see the SNMP packets. So I dont think there is an issue with the network.

It looks like the SNMP Trap Service receives the Traps but doesnt forward them onto HP SIM, or SIM isnt processing them.

How are SNMP traps sent to HPSIM? Does HPSIM listen to the SNMP Trap Service?

I have setup HP SIM to "Accept unregistered events" in the hope to see any events show up on the console.

As it stands the only events that do seem to show up are login failures to the HP SIM.

[Rob Buxton]

From a remote server that is managed by HPSIM, the Trap Destination needs to be set.
This destination is the HPSIM Server.
In addition it uses a Community Name, typically the Read Only one, public by default.

You might also want to set SNMP Service on the HPSIM Server to Allow All Hosts.

Using the HP Management Agents GUI to send a trap should generate an event.
If it is not, then I'd say there's something wrong with the SNMP setup.
As far as I'm aware, HPSIM just uses the SNMP Trap Service, that should also be running, it's typically set to manual and I think, started by HPSIM.

So, again check you're using the same community names at both the HPSIM and remote Server end, and they're case sensitive.

[deceit]

Thanks for your repsonse Rob.

I had set them up correctly, but just went in and removed the "Accepted Community Names" from both machines and readded them.

Now it works!! I have no idea why this resolved it, I am positive there wasnt a typo.

Any ideas on what causes two notifications per trap?

This happens for Disk notificaitons as well as the HP Test Trap.


Thanks

[tmagee]

Good result, that you're now getting alerts. As regards the duplicate alerts, I used to get this when I had two community names set on both the SIM server (under Options, Protocol Settings, Global Protocol Settings) and the unit being monitored (snmp service properties, Traps, Community name). I'd the same ip set for both names which corresponded with the ip of the SIM server.

Try checking those settings on your setup and have only one community name assigned to an ip on the unit being monitored and use this same community name on SIM.

Are you also now getting disk threshold alerts ok? - for rising and falling alerts? (I've not verified these yet with agents 7.4).

[deceit]

I removed one of the traps and now only get one alert which is much better. Could you tell me if I should have left the READ CREATE trap or the READ ONLY trap? Or doesnt it matter?

The rising threshold notification seems to work fine and will tell me each day if the system is still over the limit.
I still havent seen a falling threshold notification to the SIM or the local event viewer yet. My guess is it still doesnt work in 7.4.

[tmagee]

I'm assuming you're referring to the Rights of the Community Name String? (in the SNMP service properties, Security tab). I have this set for Read Only.

I remember reading somewhere about this and relates to restricting access to your systems' data - the community name acts as a password and I believe if this is known as well as your ip, then data alerts can be viewed. Keeping it at Read Only puts restrictions on this