Strange event from ESX host forwarded to SIM

[Tom Wolf_3]

Hello.

We have a several ESXi 5.5 hosts that are monitored by HPSIM (Systems Insight Manager 7.5.0 - Windows).

Periodically they report the following event through HPSIM:

HPSIM - (WBEM) Consolidated Status Collection operational status has transitioned to Error -
Event description: Consolidated Status Collection operational status has transitioned to Error -
Description: Consolidated Status Collection operational status has transitioned to error.(Collection status changed from 2 to 6) -
Probable Cause: Other -
Probable Cause Desc: Operational Status of participating members in this collection have transitioned to Error state and caused operational status for the for the collection to transition to Error

I'm totally clueless on what this event indicates.

All seems fine on the ESX host itself.

Does anyone know excatly what this event means and how to resolve?

Any assistance would be greatly appreciated.

Thank you.

[wreigle]

I am seeing the same issue. Unfortunately no idea what's causing it.

[Andrew_Haak]

My best guess would be a datacollection problem for your ESX servers, normaly datacollection is sheduled bi weekly, so the events shoild only happen one every 2 weeks arround the time the job runs, there also is a log for this job you can look at or trigger the datacollection from the menu under option when you selected the ESX servers that you have these problems with.

Sometimes an identify gives more insight as the logging for that job is more complete. If you see errors in the log you could post them here for troubleshooting.

[Tom Wolf_3]

Hello Andrew.

Thanks for responding.

We ran Data Collection against one of our ESX hosts generating the "HPSIM - (WBEM) Consolidated Status Collection..." SIM events.

Unfortunately, there was no output.

Next, we ran Identify Systems... against the ESX host.

Below is the output.

Again, unfortunately no errors so nothing really to troubleshoot.

Please let me know if you have any other suggestions.

Thanks again.

********* Significant issues and overall target discovery/identification
          status:
Normal:   The system has valid hardware data (model, serial number, and
          unique identifier) and has been identified properly for SIM to
          manage.
Normal:   System(s) have been successfully identified.

********* Additional results (listed for information only):

        * Starting identification process...
        * Checking for known running web servers...
        * Checking for SNMP protocol support on system...
Normal:   The system did not respond to SNMP. Verify security settings and
          community strings in Options->Security->Credentials->System
          Credentials
        * Checking for SNMPv3 protocol support on system...
Normal:   The system did not respond to SNMPv3. Verify security settings and
          v3 credentials in Options->Security->Credentials->System
          Credentials
        * Running HP ProLiant management agent identification...
Normal:   The system does not support SNMP.
        * Running SNMP base cluster identification using common cluster
          MIB...
Normal:   SNMP is not supported or globally disabled, skipping SNMP cluster
          identification.
        * Checking for WBEM protocol support on system...
Normal:   This system has WBEM protocol support.
Normal:   This system has at least one SMI-S CIMOM installed to serve as the
          storage proxy to discover storage systems.
Normal:   This system has a server CIMOM installed.
        * Running WBEM rules based identification...
Normal:   Got the system information from the WBEM / WMI providers and it has
          a valid WBEM credential.
        * Running Virtual Machine (VM) Host (non HP Integrity VM Host)
          Identification...
Normal:   The system is a Virtual Machine (VM) Host (non HP Integrity VM
          Host) and has (8) guests running on it.
        * Running Storage identification...
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * Storage identification completed.
        * Running HP NetServer identification...
Normal:   The system does not support SNMP.
        * Running HP-UX SNMP identification...
Normal:   The system does not support SNMP.
        * Running OA Identifier which will discover systems in an enclosure
          if the automatic enclosure discovery option for OA is enabled...
Normal:   A non-embedded operating system is detected so skipping the OA
          identification.
        * Running XMLReply Identifier which will discover systems in a
          rack/enclosure if the automatic discovery option for the container
          node is enabled...
Normal:   A non-embedded operating system is detected so skipping the
          Management processor identification.
        * Running iLO Identifier which will discover the server if the
          automatic server discovery option for iLO is enabled...
Normal:   A non-embedded operating system is detected so skipping the ILO
          identification.
        * Running p-Class Identifier...
Normal:   The system is not a p-Class so skipping p-Class identification.
        * Running IPMI protocol identification...
Normal:   Skipping IPMI discovery in favor of WBEM/SNMP discovery.
        * Running SSH Protocol identification...
Minor:    The system(s) do not have Secure Shell (SSH) installed or it is not
          running on the managed system(s).
        * Running SSH Identification...
Normal:   SSH Protocol is not enabled so skipping SSH Identification.
        * Running WS-Man identification...
Normal:   A non-embedded operating system is detected so skipping the WS-MAN
          identification.
        * Running Monarch OA Identification...
Normal:   The system is not a monarch OA so skipping monarch OA
          identification.
        * Checking for System Management Homepage and other HP web agents...
Normal:   The System Management Homepage is not installed on this system. It
          may not be supported.
        * Running HP Serviceguard Identification...
Normal:   The system is not part of HP Serviceguard cluster.
        * Running SNMP System Type manager ...
Normal:   The system does not support SNMP.
        * Running SNMP subtypes identification...
        * Resolving DNS names...
Normal:   The fully qualified domain name was successfully resolved for the
          system.
        * Checking if the Node is a Virtual Center...
Normal:   Skipping Virtual Center Identifier.
        * Running HPE Synergy identification...
        * Running Virtual Machine Management (VMM) Plug-in Identification...
        * Building system relationships...
Minor:    System Management Homepage (SMH) is not installed or is not
          responding on the system(s).
        * Running system Manage Communication diagnosis...
        * System identification finished on system.

[Andrew_Haak]

If i look at the log i see the SMH is not discovered. Is this a Gen8 or newer server or older?

[Tom Wolf_3]

Hello again Andrew.

This particular host is a BL490c G6.

With the exception of one BL680c G5, all the ESX hosts generating these events seem to be BL490c G6.

I'm told by our VMWare guys that they're running the latest SPP supported for their respective hardware platform.

Please let me know if you have any other thoughts on this.

Thanks.

[Andrew_Haak]

Could be that the firewall is blocking port 2301 and 2381, you can test this by accessing the SMH page by using the servers name like https://servername:2381 and see if you get a login page. Is the event you get for all servers around the same time once every 2 weeks or more often ?

[Tom Wolf_3]

Andrew, it doesn’t seem like SMH page is accessible on the ESX hosts I tried.

Both port 2301 and 2381 timeout.

The ILO page is reachable.

As for the timing of the events, they don’t seem to follow any type of pattern.

I’m not certain if they’ve been appearing every two weeks.

I don't believe they appear on a consistent basis.

Below is a sampling of all the events I could find in SIM.

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost22     7/24/16 8:59 PM

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost33     6/26/16 10:37 AM

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost13     5/24/16 10:42 AM

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vssjhepic103   5/12/16 9:44 AM

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  10.6.0.40      5/2/16 6:13 PM

[Andrew_Haak]

I see VSW servers as virtuals i guess , if these are the virtual servers with the errors there could be a problem with the credentials used for discovery of your virtual servers on the ESX server. Do you want virtual servers in SIM ?

It's just an assumption that these are virtuals, can you confirm this?

[Tom Wolf_3]

Hello Andrew.

Servers vswfhost22, 33, 13, and vssjhepic103 are all physical ESX hosts.

IP address 10.6.0.40 (hostname vswfhost36) is also a physical ESX host.

For the most part, we don't have the guests defined in SIM.

A few are but they have a system type of "Unknown".

These " (WBEM) Consolidated Status Collection operational status has transitioned to Error " events are only being generated on the ESX physical hosts - no guests.

They're more of a nuisance event but it would be nice to know the exact reason why they're generated.

Not sure if we're concerned enough to open a support case at this point.

Thanks.