topic Re: adding a pacemaker cluster to HP SIM 7.4 in Server Management - Systems Insight Manager

adding a pacemaker cluster to HP SIM 7.4

blentes — Thu, 29 Jun 2017 16:20:00 GMT

Hi,
first: sorry for crossposting. I asked already in another forum, but found this one later. But here my question is much better placed. Maybe someone (the morderator ?) can delete the old posting in the insight remote supprt forum, i didn't find a solution to delete my own posting (the internet will never forget ...).

My question:

We have several HP servers running, and a HP FC SAN. I currently established a two node cluster with pacemaker on SLES 11 SP4. Pacemaker has a resource agent "ClusterMon" which is intended to be used for monitoring the cluster via SNMP. My SIM is already running fine and monitoring several devices.
What i did already:
I configured SNMP. I created on both nodes a resource for the ClusterMon agent. I added pcmk-mib.mib and linux-ha-mib.mib from the pacemaker package to SIM using mcompile and mxbin. They are both listet now with mxmib -l. I added one node to SIM. I have a successfull initial data collection from it and it is shown in "Events" as "Discovered system". When i mgrate e.g. a resource, several smnp packets are sent to the SIM server (checked with tcpdump). But i see nothing in events. What do i have to do that messages from the agent are shown in "Events" and i can configure SIM furthermore that i get e-Mails if something relevant happens in the cluster.

Furthermore:
Choosing Options ==> Events ==> SNMP ==> PCMK-MIB.mib i see no traps.
Also "mxmib -t PCMK-MIB.mib" shows no traps. It seems my mib file is broken ...

Next try:
There is another possibility to monitor a pacemaker cluster with SNMP:
https://github.com/sys4/pacemaker-snmp . Everything on the side of the node of the cluster went fine.
But again "mxmib -t" for the two included mib files show no traps. I think that's the problem.
HP SIM seems to not like the mib files.
mcompile throws a lot of warnings:

vm59093-4:/opt/mx/mibs # /opt/mx/bin/mcompile -d /opt/mx/mibs/ PACEMAKER.mib

HP Systems Insight Manager Mib Compiler, Version 1.0
Copyright (c) 2003 Hewlett Packard Corporation

Compiling PACEMAKER.mib

Line: 86: Warning: Expected comma, found STRING
Line: 86: Warning: Skipping unexpected comma ,
Line: 87: Warning: Expected comma, found Integer32
Line: 87: Warning: Skipping unexpected comma ,
Line: 88: Warning: Expected comma, found Integer32
Line: 91: Warning: Expected comma, found sys4PcmkNodeIndex
Line: 92: Warning: Expected comma, found SYNTAX
Line: 93: Warning: Expected comma, found MAX-ACCESS
Line: 94: Warning: Expected comma, found STATUS
Line: 95: Warning: Expected comma, found DESCRIPTION
Line: 96: Warning: Expected comma, found index
Line: 96: Warning: Expected comma, found the
Line: 96: Warning: Expected comma, found of
Line: 96: Warning: Expected comma, found pacemaker
Line: 97: Warning: Expected comma, found ::=
Line: 97: Warning: Expected comma, found sys4PcmkNodeEntry
Line: 200: Warning: Expected comma, found STRING
Line: 200: Warning: Skipping unexpected comma ,
Line: 201: Warning: Expected comma, found INTEGER
Line: 201: Warning: Skipping unexpected comma ,
Line: 202: Warning: Expected comma, found Integer32
Line: 205: Warning: Expected comma, found sys4PcmkResourceIndex
Line: 206: Warning: Expected comma, found SYNTAX
Line: 207: Warning: Expected comma, found MAX-ACCESS
Line: 208: Warning: Expected comma, found STATUS
Line: 209: Warning: Expected comma, found DESCRIPTION
Line: 210: Warning: Expected comma, found index
Line: 210: Warning: Expected comma, found the
Line: 210: Warning: Expected comma, found of
Line: 210: Warning: Expected comma, found pacemaker
Line: 211: Warning: Expected comma, found ::=
Line: 211: Warning: Expected comma, found sys4PcmkResourceEntry
Processing import file: SNMPv2-SMI.mib
Processing import file: SNMPv2-TC.mib
Processing import file: sys4-mib.mib
Mib Compilation completed successfully

Compilation for SYS4.mib went successfully.

Can anyone give me a clue what is wrong with the mib files and what i can do ? I'm not an expert with the syntax of these files.

Thanks.

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:53:17 GMT

HI,
i made some slight changes to the PACEMAKER.mib, so compilation is going now fine without complaints. But i still don't see neither traps in the web interface of the sim nor with mxmib -t.

/opt/mx/bin/mxmib -t PACEMAKER.mib
SHOW TRAPS FOR A MIB
COMMAND COMPLETED SUCCESSFULLY

Adding the file to sim with mxmib -a also went fine.
What the hell is with this mib file ?

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

Dennis Handly — Wed, 06 Sep 2017 22:55:01 GMT

>Maybe someone (the moderator?) can delete the old posting in the insight remote support forum

You can delete posts that don't have replies by using Options > Delete, starting on your last reply.

Since I replied, you'll have to ask the moderator to delete the whole topic.

https://community.hpe.com/t5/notifications/notifymoderatorpage/message-uid/6969447

Or edit your post to nothing and I'll delete my reply and you can delete your original post.

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:56:22 GMT

Hi,
i found out something more. I checked all the mib files which are imported in the SIM using mxmib -l.
All the mib's i got i checked with mxmib -t. And i got a lot which didn't show any trap.
That's the same with the tow mib files i try to import: PACEMAKER.mib and PCMK-MIB.mib.
Checking them with mxmib -t shows no traps.
So i think the error is not a malfunction in the compilation, but just before. The syntax of the mib file is not correct for SIM.
It does not recognize any trap in the file.
Is there a tutorial or s.th. else which shows the correct syntax for mib's so that they fit the needs for SIM ?

Thanks.

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:56:49 GMT

i'm getting desperate. Importing a MIB does not seem to be a complex procedure, but in my case it is.
I found a new version of my respective mib:
https://github.com/ClusterLabs/pacemaker/blob/master/extra/PCMK-MIB.txt
Compiling and importing went without errors, and now i see ONE trap in the SIM and also with mxmib -t.
But the OID's transfered with SNMP don't match this trap, so they are not recognized and i don't see them under "Events".
Is there a tutorial or s.th. like that how a MIB should look like to be successfully imported into SIM ?
I updated SIM to 7.6, but that didn't helped.
Thanks for any help.

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:57:17 GMT

still fighting :-)). Some questions i stumbled across: which snmp versions does SIM support ? Currently i'm trying SIM 7.6.
Does SIM recognizes the protocol version automatically from the trap ? It should, it's inside the packet.
Do i have to say SIM "this device is SNMP 1, this device is SNMP2 ... ? If yes where do i have to configure it ?

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:57:51 GMT

some other news:
Options ==> Events ==> Event Filter Settings allows me to accept unregistered events. Now i see the traps at least in "Events". Investigating the events more exactly the OID is partially recognized correctly: 1.3.6.1.4.1.32723. But the details tell something about "Generic Trap ID" and "Specific Trap ID", which is, IIRC correctly, related to SNMP v1. But my packets (checked with tcpdump and wireshark, tools i trust) are SNMP v2. And the traps have the OID 1.3.6.1.4.1.32723.1.x. Is it possible that the packets are not recognized correctly ? HP SIM says the MIB is not yet compiled (which i did !), but on the other hand says under "Trap details" the associated MIB file for this trap is cpqgen.mib !?! But investigating cpqgen.mib i see it's a different OID (enterprises 232), but my PACEMAKER-MIB is enterprises 32723. OMG

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:58:19 GMT

I made a big step. In the User guide for SIM, in chapter "Compiling and customizing MIBs" i found the following:
"The mcompile tool verifies the syntax of all MIBs to be loaded into the system. mcompile resolves
all MIB dependencies and, where necessary, converts SNMP v2/v3 MIBs into v1 format for loading
into the HP SIM database". Ah, mcompile converts my SMNPv2 MIB into a SNMPv1. Now i understand why SIM always was talking of Generic Trap ID and Specific Trap ID. My agents sends v2 packets, which don't know generic and specific trap id, this is related only to v1. But SIM imports a v1 mib (after compiling), so it's difficult that the traps and SIM map.
I investigated further the SQL statements which occur when importing the cfg and when the traps arrive. And i found a small, but important difference:
This is the statement when importing the cfg file with mxmib:
INSERT INTO trapItems (trap_id, trap_name, trap_enterprise, trap_generic_id, trap_specific_id, trap_severity_id, trap_varbind_count, cat_id, sev_vb) VALUES (5359,'pacemakerNotificationTrap','1.3.6.1.4.1.32723.2',6,1,100,9,1,0).
The OID is entered as ==> 1.3.6.1.4.1.32723.2 <== for the traps from pacemaker.
But when the traps arrive SIM looks for:
SELECT trapItems.trap_id, RTRIM(trap_name), enabled_flag, trapItems.sev_vb FROM trapItems,trapItemsEx WHERE trapItems.trap_id = trapItemsEx.trap_id AND trap_generic_id = '6' AND trap_enterprise = '1.3.6.1.4.1.32723' AND trap_specific_id = '1'.
This can never match: Entering 1.3.6.1.4.1.32723.2 as trap_enterprise, but looking for ==> 1.3.6.1.4.1.32723 <==.
I made some sligth changes to the PCMK-MIB.cfg file:
vm63422-1:/opt/mx/mibs # diff PCMK-MIB.cfg.orig PCMK-MIB.cfg
227,228c227,228
< ENTERPRISE pacemakerTrap
< VARIABLES { pacemakerNotificationNode, pacemakerNotificationResource, pacemakerNotificationOperation, pacemakerNotificationDescription, pacemakerNotificationStatus, pacemakerNotificationReturnCode, pacemakerNotificationTargetReturnCode, pacemakerNotificationAttributeName, pacemakerNotificationAttributeValue }
---
> ENTERPRISE pacemaker
> VARIABLES { pacemakerNotificationResource, pacemakerNotificationNode, pacemakerNotificationOperation, pacemakerNotificationDescription, pacemakerNotificationReturnCode, pacemakerNotificationTargetReturnCode, pacemakerNotificationStatus, pacemakerNotificationAttributeName, pacemakerNotificationAttributeValue }
229a230,232
> --#SEVERITY MINOR
> --#CATEGORY Pacemaker
> --#MSG_FORMATTER "$V1V#Resource: # $V2V#Node: # $V3V#Operation: # $V4V#Description: # $V5V#Returncode: # $V6V#TargetReturncode: # $V7V#Status: # $V8V#AttributName: # $V9V#AttributeValue: #"

The change in ENTERPRISE is necessary so that the OID (trap_enterprise) is entered in the db as 1.3.6.1.4.1.32723, so that it matches with the trap recognition.
SEVERITY and CATEGORY help that the traps are directly configured as i want.
And MSG_FORMATTER is helpful that an e-Mail which is related to this event is a bit more informative.

Hopefully this helps someone when stumbling across the same problem.

Bernd

Re: adding a pacemaker cluster to HP SIM 7.4

blentes — Wed, 06 Sep 2017 22:58:50 GMT

supplement: i used this MIB file:
https://github.com/ClusterLabs/pacemaker/blob/master/extra/PCMK-MIB.txt

Bernd