Systems Insight Manager
cancel
Showing results for 
Search instead for 
Did you mean: 

Strange event from ESX host forwarded to SIM

Tom Wolf_3
Valued Contributor

Strange event from ESX host forwarded to SIM

Hello.

We have a several ESXi 5.5 hosts that are monitored by HPSIM (Systems Insight Manager 7.5.0 - Windows).

Periodically they report the following event through HPSIM:

HPSIM - (WBEM) Consolidated Status Collection operational status has transitioned to Error -
Event description: Consolidated Status Collection operational status has transitioned to Error -
Description: Consolidated Status Collection operational status has transitioned to error.(Collection status changed from 2 to 6) -
Probable Cause: Other -
Probable Cause Desc: Operational Status of participating members in this collection have transitioned to Error state and caused operational status for the for the collection to transition to Error

I'm totally clueless on what this event indicates.

All seems fine on the ESX host itself.

Does anyone know excatly what this event means and how to resolve?

Any assistance would be greatly appreciated.

Thank you.

 

17 REPLIES
wreigle
Occasional Advisor

Re: Strange event from ESX host forwarded to SIM

I am seeing the same issue. Unfortunately no idea what's causing it.

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

My best guess would be a datacollection problem for your ESX servers, normaly datacollection is sheduled bi weekly, so the events shoild only happen one every 2 weeks arround the time the job runs, there also is a log for this job you can look at or trigger the datacollection from the menu under option when you selected the ESX servers that you have these problems with.

 

Sometimes an identify gives more insight as the logging for that job is more complete. If you see errors in the log you could post them here for troubleshooting.

Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello Andrew.

Thanks for responding.

We ran Data Collection against one of our ESX hosts generating the "HPSIM - (WBEM) Consolidated Status Collection..." SIM events.

Unfortunately, there was no output.

Next, we ran Identify Systems... against the ESX host.

Below is the output.

Again, unfortunately no errors so nothing really to troubleshoot.

Please let me know if you have any other suggestions.

Thanks again.

 

 

********* Significant issues and overall target discovery/identification
          status:
Normal:   The system has valid hardware data (model, serial number, and
          unique identifier) and has been identified properly for SIM to
          manage.
Normal:   System(s) have been successfully identified.

********* Additional results (listed for information only):

        * Starting identification process...
        * Checking for known running web servers...
        * Checking for SNMP protocol support on system...
Normal:   The system did not respond to SNMP. Verify security settings and
          community strings in Options->Security->Credentials->System
          Credentials
        * Checking for SNMPv3 protocol support on system...
Normal:   The system did not respond to SNMPv3. Verify security settings and
          v3 credentials in Options->Security->Credentials->System
          Credentials
        * Running HP ProLiant management agent identification...
Normal:   The system does not support SNMP.
        * Running SNMP base cluster identification using common cluster
          MIB...
Normal:   SNMP is not supported or globally disabled, skipping SNMP cluster
          identification.
        * Checking for WBEM protocol support on system...
Normal:   This system has WBEM protocol support.
Normal:   This system has at least one SMI-S CIMOM installed to serve as the
          storage proxy to discover storage systems.
Normal:   This system has a server CIMOM installed.
        * Running WBEM rules based identification...
Normal:   Got the system information from the WBEM / WMI providers and it has
          a valid WBEM credential.
        * Running Virtual Machine (VM) Host (non HP Integrity VM Host)
          Identification...
Normal:   The system is a Virtual Machine (VM) Host (non HP Integrity VM
          Host) and has (8) guests running on it.
        * Running Storage identification...
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * This system has some storage related information and additional
          storage systems may be added.
        * Storage identification completed.
        * Running HP NetServer identification...
Normal:   The system does not support SNMP.
        * Running HP-UX SNMP identification...
Normal:   The system does not support SNMP.
        * Running OA Identifier which will discover systems in an enclosure
          if the automatic enclosure discovery option for OA is enabled...
Normal:   A non-embedded operating system is detected so skipping the OA
          identification.
        * Running XMLReply Identifier which will discover systems in a
          rack/enclosure if the automatic discovery option for the container
          node is enabled...
Normal:   A non-embedded operating system is detected so skipping the
          Management processor identification.
        * Running iLO Identifier which will discover the server if the
          automatic server discovery option for iLO is enabled...
Normal:   A non-embedded operating system is detected so skipping the ILO
          identification.
        * Running p-Class Identifier...
Normal:   The system is not a p-Class so skipping p-Class identification.
        * Running IPMI protocol identification...
Normal:   Skipping IPMI discovery in favor of WBEM/SNMP discovery.
        * Running SSH Protocol identification...
Minor:    The system(s) do not have Secure Shell (SSH) installed or it is not
          running on the managed system(s).
        * Running SSH Identification...
Normal:   SSH Protocol is not enabled so skipping SSH Identification.
        * Running WS-Man identification...
Normal:   A non-embedded operating system is detected so skipping the WS-MAN
          identification.
        * Running Monarch OA Identification...
Normal:   The system is not a monarch OA so skipping monarch OA
          identification.
        * Checking for System Management Homepage and other HP web agents...
Normal:   The System Management Homepage is not installed on this system. It
          may not be supported.
        * Running HP Serviceguard Identification...
Normal:   The system is not part of HP Serviceguard cluster.
        * Running SNMP System Type manager ...
Normal:   The system does not support SNMP.
        * Running SNMP subtypes identification...
        * Resolving DNS names...
Normal:   The fully qualified domain name was successfully resolved for the
          system.
        * Checking if the Node is a Virtual Center...
Normal:   Skipping Virtual Center Identifier.
        * Running HPE Synergy identification...
        * Running Virtual Machine Management (VMM) Plug-in Identification...
        * Building system relationships...
Minor:    System Management Homepage (SMH) is not installed or is not
          responding on the system(s).
        * Running system Manage Communication diagnosis...
        * System identification finished on system.

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

If i look at the log i see the SMH is not discovered. Is this a Gen8 or newer server or older?
Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello again Andrew.

 

This particular host is a BL490c G6.

With the exception of one BL680c G5, all the ESX hosts generating these events seem to be BL490c G6.

I'm told by our VMWare guys that they're running the latest SPP supported for their respective hardware platform.

Please let me know if you have any other thoughts on this.

Thanks.

 

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

Could be that the firewall is blocking port 2301 and 2381, you can test this by accessing the SMH page by using the servers name like https://servername:2381 and see if you get a login page. Is the event you get for all servers around the same time once every 2 weeks or more often ?

Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Andrew, it doesn’t seem like SMH page is accessible on the ESX hosts I tried.

 

Both port 2301 and 2381 timeout.

 

The ILO page is reachable.

 

As for the timing of the events, they don’t seem to follow any type of pattern.

 

I’m not certain if they’ve been appearing every two weeks.

 

I don't believe they appear on a consistent basis.

 

Below is a sampling of all the events I could find in SIM.

 

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost22     7/24/16 8:59 PM

                      

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost33     6/26/16 10:37 AM

                      

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vswfhost13     5/24/16 10:42 AM

                      

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  vssjhepic103   5/12/16 9:44 AM

                      

        Not cleared    Major   Consolidated Status Collection operational status has transitioned to Error  10.6.0.40      5/2/16 6:13 PM

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

I see VSW servers as virtuals i guess , if these are the virtual servers with the errors there could be a problem with the credentials used for discovery of your virtual servers on the ESX server. Do you want virtual servers in SIM ?

It's just an assumption that these are virtuals, can you confirm this?

Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello Andrew.

Servers vswfhost22, 33, 13, and vssjhepic103 are all physical ESX hosts.

IP address 10.6.0.40 (hostname vswfhost36) is also a physical ESX host.

For the most part, we don't have the guests defined in SIM.

A few are but they have a system type of "Unknown".

These " (WBEM) Consolidated Status Collection operational status has transitioned to Error " events are only being generated on the ESX physical hosts - no guests.

They're more of a nuisance event but it would be nice to know the exact reason why they're generated.

Not sure if we're concerned enough to open a support case at this point.

Thanks.

 

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

Hello again,

 

i looked at the log and this is a identify log not a data collection log, i will look if you can see logging for datacollection. The only things that remain are the SSH and SMH errors, if you look at the credentials stored  for this server do you see SSH and/or trust credentials?

 

I'll have a look at the datacollevtion logging saying more than ok.

Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello Andrew.

As mentioned previously, data collection doesn't return anything.

Below is the data collection output for two of our ESX hosts.

Please note standard output and standard error are "None".

Credentials used are the same for the ESX hosts - Access Type: wbem, Credential: root, Source: Discovery (EXSI), OK to Try Others: No.

 

 ID: 133084

Status:  
  Complete
  Start time:  8/3/16 12:15 PM
Task name:  Data Collection  End time:  8/3/16 12:16 PM
Tool:  Data Collection  Duration:  46 sec
Owner:  WFSI\wfcim  Target:  vswfhost22
Run by:  WFSI\wfcim
Command:  N/A
   System Name:  vswfhost22
   Status:  
  Complete
   Exit code: N/A
   Stdout: None
   Stderr: None

 

 ID: 133106

Status:  
  Complete
  Start time:  8/3/16 12:18 PM
Task name:  Data Collection  End time:  8/3/16 12:19 PM
Tool:  Data Collection  Duration:  35 sec
Owner:  WFSI\wfcim  Target:  vswfhost33
Run by:  WFSI\wfcim
Command:  N/A
   System Name:  vswfhost33
   Status:  
  Complete
   Exit code: N/A
   Stdout: None
   Stderr: None

 

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

Hello again,

That's what i mean by no logging, i believe the logging can be set higher and then you can find the logging in the logs folder in the SIM installation folder, currently i'm not anywhere near a SIM. Server, monday is the first day i'm can check for you.
Kind regards,

Andrew
Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Okay.

Thanks Andrew.

Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello again Andrew.

 

Did you have any more info on this?

 

Thanks for all your help so far.

n0kia
Valued Contributor

Re: Strange event from ESX host forwarded to SIM


One of the classes is reporting the error.

If you SSH into your ESXi host and run the following command

enum_instances SMX_ComputerSystem root/HPQ

Look for this line that says: "Operational Status"
Example in my lab: OperationalStatus = { 6, 2, 6, 2, 2, 2, 2, 2, 2, }

StatusDescriptions = { One or more of the subsystems that make up this computer system have an Operational status value of Error, OK, Error, OK, OK, OK, OK, OK, OK, }

In my setup it is reflected upon the following below status components in order, 6 is error for SMX_ComputerSystem, 2 for SMX_BladeEnclosureCollection, 6 for SMX_EthernetCollection etc etc.
{SMX_ComputerSystem,SMX_BladeEnclosureCollection,SMX_EthernetCollection,SMX_FanCollection,SMX_MPCollection,SMX_MemoryCollection,SMX_ProcessorCollection,SMX_SAStorageSpecificCollection,SMX_SensorCollection}


So the 6 error status is Error in my SMX_EthernetCollection.

If you run the command enum_instances SMX_EthernetCollection root/HPQ

You will now see the error reflected for that status and you will need to investigate on your end why, for me it was because it's not plugged in. The SIM Health status should reflect that it's in a major status which it is.

[root@SERVER:~] enum_instances SMX_EthernetCollection root/HPQ

SMX_EthernetCollection.InstanceID="HPQ:SMX_EthernetCollection-1"
InstanceID = HPQ:SMX_EthernetCollection-1
GroupStatusDescriptions = { Error, }
GroupOperationalStatus = { 6, }
Caption = Ethernet Adapter Collection
Description = This is a collection of all Ethernet Ports.
ElementName = Ethernet Adapter Collection
[root@SERVER:~]


0(Unknown)
1(Other)
2(OK)
3(Degraded)
6(Error)
10(Stopped)

Also there is no SMH modules for Esxi so you wouldn't be able to access it 2381/2301

Tom Wolf_3
Valued Contributor

Re: Strange event from ESX host forwarded to SIM

Hello and thanks for responding.

 

Let me pass that information along to our VMWare admin's and see what they say.

 

Thanks again.

Andrew_Haak
Honored Contributor

Re: Strange event from ESX host forwarded to SIM

Thans for the help Nokia, if there is a nic or any other error it should be visible in the status right?

And you are correct that there should not be a SMH. These became obsolete when they stopped with the SNMP agents and switched to WBEM.

I've never seen these errors since we use Gen8 and AMS for a long time.
Kind regards,

Andrew