WBEM Polling is working, but No Events/alerts are being created/emailed on monitored VM Host

[DaveLewis]

First let me start by saying I'm totally green when it comes to SIM.  I have it installed on a Windows server for the sole purpose of monitoring some of the critical hardware components like power supplies and hard drives etc.  on my VMware esxi 5.5.0 servers.

I have SIM running.. I have the WMI Mapper Proxy running.

SIM discovered both the ILO and the VM host, as well as the vm machines running on the host, and when it did i did get email notification of those events.

I ran through the subscribe WBEM events which reported it was succesful.  I believe I have all the required hp offline packages installed on the VM Host. 

under the mangement protocols on the host I does show WBEM 
WBEM:1.0, SSH:SSH-2.0-OpenSSH_5.6

When I pull a power supply, the Health status eventually updates with a minor failure but at no point does the even show up in the events section and of course I also do not get notified by email. 

I have restarted both the SIM server, as well as the VM Host several times, I have deleted and rediscovered the box a couple of times. 

after a couple of days beating my head against the desk I'm at a loss.  Does anyone have any suggestions/ideas.  Is this something that is a normal stupid newbie mistake ?

Thanks in advance for any help!

Dave

[BPE]

Your ESXi firewall blocks the WBEM indication to your SIM server. Enable the NFS client rule or build your own rule and it should work.

[DaveLewis]

Enabled NFS Client (which just for detailed sake is a tcp from any 0-65535), and retested, unfortunetly still the same condition.  I see the failure after a period of time in the System Health in SIM , but not in events for that host.and thus no emails and such.

Dave

[Andrew_Haak]

Hello there,

Do you get any events from the others servers ? Could it be that the Firewall on your SIM server is blocking the WBEM traps ?

Kind regards,

Andrew

[BPE]

If it is to long ago that you started with your test the ESXi WBEMProvider stack is blocked. Or even worse the CIMMON  Server service failed. I would try to restart (better use stop and afterwards start) the service and check inside SIM on the system page of the esxi host if you can access the WBEM information. You should see NIC Memory Storage information. It takes a certain time until the WBEM information is read from the CIMMON on the ESXi.

I would also check if the subscription still exists. If you use Insight Remote Support it could happen that a time filter blocks your event.

[BPE]

Good imput from Andrew if you want to see the WBEM indication inside SIM and you have no IRS installed your pegasus WMI Mapper Indication Service  should run which is used for receiving the WBEM Indication for SIM. Sometimes a problem if you used a doamin admin for installation. If the service are not startable try to set the service login account to the user you used for installation.

[Tushar Bajpai]

Hi,

you are right if the event is not shown in the All Event, Email will not be sent. Now there are couple of steps that you have to try:

1) ESXi Image (From HP or Plain Vanilla ESXi), if it is directly from VMware you have to check for teh Providers on ESXi.

esxcli software vib list | grep hp-smx

Providers are must for getting Indication (WMI Mapper plays no role in case of ESXi, it is only for Managing Windows Box).

Get the providers and install them.

2) ReIdentify first after installing the providers. If already installed Go to -> option -> Subscribe WBEM indication

you can use CLI mxwbemsub to do the same.

3) Now After WBEM Subscription Successful, try the Test again.

[DaveLewis]

First of all let me thank all of your for your time and comments.

Let me try and reply to each one.

Andrew : Unfortunetly this is the only box that I'm trying to monitor at this point.  I was sort of using it as a test bed to see how well things were going to work.  all the windows firewalls are disabled on the SIM box.

BPE: I have tried restarting the CIM Server on the esxi host.  from the SIM I am able to see the WBEM information (the subsystems of memory, proc etc etc) .  I don't believe that I have insite remote support ? I'm assuming since the WMI mapper services are running that the account shouldn't be an issue ?

Tushar : I am using a Vanilla ESXi install.  I believe I have all the required components ?

~ # esxcli software vib list | grep hp-smx
hp-smx-provider                500.03.03.20.2-434156                 Hewlett-Packard  VMwareAccepted    2014-01-03
~ #

I have tried resubscribing to the WBEM events again and still no luck.  I will post some screen captures in a few posts to see if anyone has any further thoughts.

Thanks again for the comments.  !

Dave

[DaveLewis]

Here are some screen captures.

[Andrew_Haak]

Hello again Dave,

when you have your ESX server selected and choose Configure > Manage Communication you should see two green checks. One for identification and one for the ability to receive events. Do you also see these both green ?

You could also do a configure / repair and select the first to option under step 3 :

1) Create subscription to WBEM events

2) Send a sample WBEM / WMI indication to this instance of HP SIM to test that events appear in HP SIM event lists.

You should then see an events in SIM. You need sign-in credentials for this task

Kind regards,

Andrew

[Tushar Bajpai]

I would like to see the output of: mxwbemsub -l <nodename>

Can you please help me with the same.

[DaveLewis]

Andrew: attached is a screen capture of my attempt to subscribe ad send events  which didn't work, as well as my mange events screen.

Tushar: I'm assuming nodename is the esxi node.  it didn't like that command.

C:\Program Files\HP\Systems Insight Manager\bin>mxwbemsub -l 10.10.10.201 --user
"LEWIS\xxxxxxx" --pass xxxxxxx

No nodes were specified.
Usage:
       mxwbemsub -a [destination] ( -n nodenames | -f filespec ) [-t indication
type]
       mxwbemsub -r [destination] ( -n nodenames | -f filespec ) [-u uuid] [-t i
ndication type]
       mxwbemsub -l [destination] ( -n nodenames | -f filespec ) [-s ] [-t indic
ation type]
       mxwbemsub -m [current destination] new destination ( -n nodenames | -f fi
lespec ) [-t indication type]

       where: indication type= health
       where: s = list all WS-Man subscriptions
       where: u = UUID of WS-Man subscription

[Tushar Bajpai]

Yes, thats correct.

you can use: "mxwbemsub -l <esxi-node-name / IP Address>"

Just try running: mxwbemsub -l 10.10.10.201

No User name and Password are required here, command will get that from the DB.

[DaveLewis]

when I do that I throws an error

C:\Program Files\HP\Systems Insight Manager\bin>mxwbemsub -l 10.10.10.201
There was a problem connecting to the HP Systems Insight Manager server. Make su
re that:
1. Your username has been added to HP Systems Insight Manager.
2. Your username and password, if specified, are correctly spelled.
3. HP Systems Insight Manager is running.
4. You used '--' for any long options and double quotes if your username include
s a domain.
Example: <commandname> --user "mydomain\myusername" --pass mypassword

C:\Program Files\HP\Systems Insight Manager\bin>

[DaveLewis]

and if I try it as a "run as administrator" cmd prompt

C:\Program Files\HP\Systems Insight Manager\bin>mxwbemsub -l 10.10.10.201

No nodes were specified.
Usage:
       mxwbemsub -a [destination] ( -n nodenames | -f filespec ) [-t indication
type]
       mxwbemsub -r [destination] ( -n nodenames | -f filespec ) [-u uuid] [-t i
ndication type]
       mxwbemsub -l [destination] ( -n nodenames | -f filespec ) [-s ] [-t indic
ation type]
       mxwbemsub -m [current destination] new destination ( -n nodenames | -f fi
lespec ) [-t indication type]

       where: indication type= health
       where: s = list all WS-Man subscriptions
       where: u = UUID of WS-Man subscription

[DaveLewis]

Ok. I don't konw what I did.. but some more reboots and now I have this when run the command from the server


C:\Program Files\HP\Systems Insight Manager\bin>mxwbemsub -l -n 10.10.10.201

Filter instances of Subscriptions for:

10.10.10.201
Filter Name: HPSIM_TYPE_1_hpmon_0
Query: select * from CIM_AlertIndication
Destination: https://10.10.10.241:50004/10.10.10.201?wbemip=10.10.10.201


and now when I reboot the vm host I get some events reporting "system unreachable" and "system reachable". but I'm and the ILO mangement side of things seems to be reporting events  but still nothing from a hardware perspective on the actual server itself. 

[Andrew_Haak]

Did you reinsert the power suply and then remove it again?

There should also be some post errors. By the way the screenshot looked great both balls where green, when you hover above the second ball it should say ability to recieve events and the time/date.

Kind regards,

Andrew

[DaveLewis]

Andrew,

I would totally agree with you but it isn't lol.

If I pull the power cable on the psu, the health status does eventually change (I'm guessing at the next polling intrerval) and If I drill down it tells me which psu is showing the fault, but at no point do I get an event, so no email either.

As I said earlier, when I reboot now, I do get some events and emails , but only if I do something at that level, at the component level I don't get anything. And yes on the communications page if I mouse over the green checkbox on events it says it's able to receive events.

I have to say, I know I'm a newbie to SIM , but it has to be one of the most complicated systems I've seen just to report system events.  lol.    All this just to try and monitor my couple of vm boxes so I don't have to physically look at them every day for some flashing orange lights. 

I'm open to suggestions if you can think of anything.

Thanks,

Dave

[Andrew_Haak]

Hello again Dave,

Do you also have Windows servers in your monitoring (can also be your SIM server)? Could you try to open the SMH page for a Windows server and do a test for WBEM event send? You can find this under Settings > Send Test Indication. Choose any eventype and look for these events in your SIM server under Events by Severity > All Events.

You can also check the WBEm proxy settings under Options > Protocol Settings > WMI Mapper Proxy. It should contain your server and the default port. The WMI proxy also has a logfile at c:\Program Files\The Open Group WMI Mapper\logs\PegasusStandard.log or the Error.log for that matter. Perhaps you can find the error there.

Kind regards,

Andrew

[MichaelHayes]

I was able to receive WBEM events after I disable the ESXi firewall. 

esxcli network firewall set --enabled false (turn firewall off)

nc -z HP_SIM_SERVER_IP 50004 

/opt/hp/tools/./hptestevent

esxcli network firewall set --enabled true (turn firewall on)

[Andrew_Haak]

That's strange,

one would think a firewall would stop incomming traffic not outgoing traffic.

[LGentile]

I'm actually having this same issue.  I am surprised more people haven't mentioned it.

It appears port 5989 is blocked for outgoing CIM requests over WBEM.  The firewall guide says it should be open by default when the CIM server is enabled, however looking at the vSphere client, I can see that there is no outgoing port enabled.

Unfortunately, I can't find any documentation on how to change the rule for this service to allow outgoing connections.  All I have found is how to limit hosts from connecting to it, but I need the WBEM indications to go outward.  The only way I have been able to get this to work is to disable the firewall completely, but that's not really something I want to do.

This didn't happen with 4.1 and I've read it's fixed in 5.5.  I can't understand why a patch hasn't fixed this behavior.  The CIM service is supposed to open the port temporarily to send out indications, according to this troubleshooting document:

http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.cimsdk.smashpg.doc/04_CIM_SMASH_PG_Troubleshooting.6.3.html

Anyhow..  still researching.  I've had to use vCenter alarms to work around this, which is really annoying when you have a lot of hosts since it's very difficult to configure alarms for only a subset of your environment (we have 4.x and 5.0 in addition to 5.1 and it's only the 5.1s that don't work correctly).

[Andrew_Haak]

Hello there,

One solution if you have G8 or G9 servers, you can enable SNMP traps on the ILO for hardware events.(Agentless monitoring)

As far as i know wbem traps use dynamic traps so the default WBEM 5989 port would not be enough.

[LGentile]

Yeah, agentless has been something I have wanted to do for a while, but we have so many Gen8 machines deployed, the work involved to get them configured for agentless is somewhat large.  Granted, I have been working with PowerShell and the new scripting cmdlets to make this easier, but it is still a big task.

I had SNMP forwarded for a while, but sometimes you get strange results (IP addresses or serial #s instead of host names, etc) so i need to test this further and ensure the settings are correct on the iLOs to support agentless properly).  I think have the issue was that passthrough was set on most of them so the traps were not as i expected them to be.

Thanks

[sundargit]

Hi, I have configured nearly thousand servers HPSIM configuration and also Test (Generic Trap) alert successful. but Many servers having hardware failures but not getting these events in HPSIM console (CMS). How to pull hardware events into HPSIM manually. i tried hardware status pulling option in hpsim console but no luck. This is my major issue, please help to resolve this