WBEM - health status RED - no events ? Win2008

[budykiller03]

Hello,

I'm lost with HP SIM, it's kind of hard to implement. Please help me understand why no events are recorded when health status change

I fully configured HP SIM to work with my ESX VMs, and i receive all the mails when events are registered. OK

However, on my other physical Windows 2008 R2 machine, it doesn't work.

- I have installed the WBEM Provider

- The system is well discovered by HP SIM.

- I can see changes on health status.

- Firewall is disable for the tests.

But whenever i see through HP SIM  that health status of the server change (i do this by activating / disabling the Second NIC), it doesn't report any events despite a minor icon showing a network problem in the health status !

Same thing when i "test WMI" on this Windows 2008 machine. There is a NT event recorded but it doesn't report to HP SIM.

I'm lost. Can someone show me the way to debug this issue ?

Thank you.

[budykiller03]

My HP SIM logs are showing weird entries.

@!@,2012-07-26 10:31:05 CEST,JOB,PROGRESS,START,JOB,41843_tlss004.mydomain.fr,VERBOSE,MYDOMAIN\administrateur,,,
    Running Tool:Subscribe to WBEM Events
    Expanded Command Line:mxwbemsub -a -n tlss001.mydomain.fr
    Targets:
        tlss004.mydomain.fr

@!@,2012-07-26 10:31:05 CEST,JOB,PROGRESS,START,JOB,41843_tlss004.mydomain.fr:tlss004.mydomain.fr,DETAIL,mydomain\administrateur,,,
    Running Tool:Subscribe to WBEM Events

@!@,2012-07-26 10:31:06 CEST,SESSION,FAILURE,LOGIN,USER,Sign-In Attempt By Invalid User (127.0.0.1),WARNING,mydomain\TLSS004$(MxpiMain5_2),,,
@!@,2012-07-26 10:31:05 CEST,JOB,SUCCESS,DONE,JOB,41843_tlss004.mydomain.fr:tlss004.mydomain.fr,VERBOSE,mydomain\administrateur,,,
    Running Tool:Subscribe to WBEM Events

 My event viewer on the HP SIM server (Win2008R2) :

Échec d’ouverture de session d’un compte.

Sujet :
	ID de sécurité :		Système
	Name account :		TLSS004$
	Domain account :		MYDOMAIN
	ID d’ouverture de session :		0x3e7

Type d’ouverture de session :			2

Compte pour lequel l’ouverture de session a échoué :
	ID de sécurité :		NULL SID
	Account name :		root
	Domain account :		TLSS004

Information about the error :
	Error detail :		unknow username or invalid password
	État :			0xc000006d
	Sous-état :		0xc0000064

Informations sur le processus :
	ID du processus de l’appelant :	0x7cc
	Nom du processus de l’appelant :	C:\Program Files (x86)\The Open Group\WMI Mapper\bin\WMIServer.exe

 The logs for the WMI mapper (PegasusStandard.txt) is full of :

07/22/12-07:23:59 WARNING cimserver: Failed to Login. Invalid username or password.Request from "192.168.0.4".

I'm really lost. What is wrong ? Why WMIserver is using the username "root" ? This user is only used in the Global Credentials for the ESX part working with Linux.

I also never had the opportunity to "subscribe to WBEM events" without having an error telling me my username is wrong. I modified the NTFS security to allow the domain admin to execute all exe in the "BIN" folder of HP SIM, but it did nothing.

The only way i can subscribe to WBEM events without an error is using the command-cli directly. Then the WBEM events are registered. However like i said in my fist post, i don't receive anything in the event view. But with all the WMIServer error, the problem seems to be here. How can i make things work ?

Thank you for your help.

[budykiller03]

Ok sorry for posting quickly, i'm just trying to help others understand my issue.

I just reload the HP SIM service, and the two Pegasus service for WMI.

I looked for new entries in the Pegasus LOG file, and i can see the server restarted, plus 3 "invalid username and password".

The NT event viewer shows many Audit Success, and 3 failed attempts. The success entries show different username, but good ones.

The 3 failed attempts occured in a few second, and with the same username "root".

My conclusion is that when you reload the server and access the SIM webpage, all username in the "Sign-in" credentials are tested with Pegasus WMI. So it doesn't work for root as it's a Linux account, but all others are working fine.

I'm a bit disappointed because i thought the problem was here.

I still cannot subscribe to WBEM event through the SIM webpage, nor i receive events.

[budykiller03]

Seriously, i'm going to think i've found a bug in this software.

1) i told you i wasn't able to create subscription to WBEM event without using the Command-cli, right ?

Now i can't create subscription anymore !

- HP SIM keep telling me "wrong user" etc.

- command-cli

C:\Program Files\HP\Systems Insight Manager\bin>mxwbemsub -a -n tlss001.mydomain.fr

Create Subscriptions for:
tlss001.mydomain.fr
FAILED to create indication subscription.

Cause: Unable to create a socket connection on the managed system

Recommended Action: Managed system must be listening on socket 5989 (WBEM) or 2381 (System Management Homepage).  If the managed system is running Windows, then ensure System Management Homepage is running.  Otherwise ensure WBEM is accessible.

 OK ok !! I've even force a reinstallation of the HP SIM Wbem providers from here :

HP WBEM Providers download page

Right now :

- WBEM still Works in HP SIM Discovery for that system. (it says WBEM fully working + i can see health status changing)

- but i can't create subscription because the socket can't be created with the manage system.

All firewalls have been taken down!

- The one on the desired managed system (Tlss001)

- The other one on the HP SIM Server (TLSS004).

Can HP explain me how HP SIM can talk to my server in WMI and see health status changing but can't create a socket with WMI to subscribe to events ?

Heeeeelp.

Thanks anyone who can help me, these are production server, this is not a joke and i've already lost 3 days to fix this by all means :(

[SwisspostIT]

hi,

I'm not using wbem myself, I use on windows Systems only SNMP but still I'll try to help.

check following:

under the systems properties of the windows system go to Tools&Links --> system credentials --> edit system credentials

then click on "Show advanced protocol credentials" and under tab "WBEM/WMI" check the credentials there or enter an account with administrative rights on the managed system.

maybe this helps but since I'm not using WBEM on Windows I'm not sure if that's the right way to do it (since the entered account could change its password, maybe there is a better way)

[budykiller03]

Thank you very much.

I already check this 3 or 4 times, the account is valid and i was able to create subscriptions before, this is just weird.

Please have a look at the attached picture. It says both credentials are used because they were able to communicate with the target.

[budykiller03]

Port mirroring from my switch to see traffic, shows many entries from HP SIM server when i try to subscribe to WBEM event.

Wireshark indicates "DCERPC" packets. I can see it uses the right user (administrateur).

I also check WMI connectivity to TLSS001 with WBEMTEST.exe from my HPSIM server.

- Connection is OK

- I ran the following request "select * from Win32_LogicalDisk" as a test, and it returns : "C,D,E,F", it's OK.

Conclusion :

- WMI is working on the target machine.

- i can access it and do requests as well.

- HP SIM server can't create subscription however.

So strange.

[budykiller03]

Logs from HP WBEM server on the target machine (tlss001) :

07/26/2012 16:59:19.066 [Error] HPWmiSys.DLL!CHPCommonLogEntryProv::CreateInstanceEnumAsync| Cannot return instances
07/26/2012 16:59:19.144 [Information] HPWmiProc.DLL!CHPProcessorCollectionProv::GetInstanceByPath| Returning instance by path
07/26/2012 16:59:19.176 [Information] HPWmiSys.DLL!CHPWinComputerSystemList::RebuildList| GetApplianceData(): .ini file does not exist at the expected location
07/26/2012 16:59:19.191 [Information] HPWmiSys.DLL!CHPWinComputerSystemProv::GetInstanceByPath| Returning instance by path
07/26/2012 17:02:18.898 [Information] HPWmiSensor.DLL!CHPWmiSensorModule::~CHPWmiSensorModule| DLL is unloaded.
07/26/2012 17:02:18.898 [Information] HPWmiMgmtProc.DLL!CHPWmiMgmtProc::~CHPWmiMgmtProc| DLL is unloaded.
07/26/2012 17:02:18.898 [Information] HPWmiMemory.DLL!CHPWmiMemoryModule::~CHPWmiMemoryModule| DLL is unloaded.
07/26/2012 17:02:18.898 [Information] HPWmiProc.DLL!CHPWmiProcModule::~CHPWmiProcModule| DLL is unloaded.
07/26/2012 17:02:18.898 [Information] HPWmiBlade.DLL!CHPWmiBladeModule::~CHPWmiBladeModule| DLL is unloaded.
07/26/2012 17:02:18.898 [Information] HPWmiSys.DLL!CHPWmiSysModule::~CHPWmiSysModule| DLL is unloaded.
07/26/2012 17:04:23.340 [Information] HPWmiSys.DLL!CHPWmiSysModule::CHPWmiSysModule| DLL is loaded.
07/26/2012 17:04:23.371 [Information] HPWmiMgmtProc.DLL!CHPWmiMgmtProc::CHPWmiMgmtProc| DLL is loaded.
07/26/2012 17:04:23.371 [Information] HPWmiMemory.DLL!CHPWmiMemoryModule::CHPWmiMemoryModule| DLL is loaded.
07/26/2012 17:04:23.371 [Information] HPWmiBlade.DLL!CHPWmiBladeModule::CHPWmiBladeModule| DLL is loaded.
07/26/2012 17:04:23.371 [Information] HPWmiProc.DLL!CHPWmiProcModule::CHPWmiProcModule| DLL is loaded.
07/26/2012 17:04:23.371 [Information] HPWmiMemory.DLL!CHPHostedMemoryCollectionList::RebuildList| Health (or MHP) driver not found
07/26/2012 17:04:23.371 [Information] HPWmiMgmtProc.DLL!CHPMPHostedCollectionProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.371 [Information] HPWmiMemory.DLL!CHPHostedMemoryCollectionProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.371 [Information] HPWmiBlade.DLL!CHPBladeEnclosureGroupHostedCollectionProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.387 [Information] HPWmiProc.DLL!CHPProcessorGroupHostedCollectionProv::CreateInstanceEnumAsync| Returning 1 instances
07/26/2012 17:04:23.403 [Information] HPWmiSensor.DLL!CHPWmiSensorModule::CHPWmiSensorModule| DLL is loaded.
07/26/2012 17:04:23.418 [Information] HPWmiSensor.DLL!CHPHostedPowerCollectionProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.418 [Information] HPWmiSensor.DLL!CHPHostedCoolingCollectionProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.434 [Information] HPWmiSensor.DLL!CHPTempGroupHostedCollProv::CreateInstanceEnumAsync| Returning 0 instances
07/26/2012 17:04:23.481 [Information] HPWmiSys.DLL!CHPIMLAccess::RebuildList| GetApplianceData(): .ini file does not exist at the expected location
07/26/2012 17:04:23.496 [Error] HPWmiSys.DLL!CHPCommonLogEntryProv::CreateInstanceEnumAsync| Cannot return instances
07/26/2012 17:04:23.559 [Information] HPWmiProc.DLL!CHPProcessorCollectionProv::GetInstanceByPath| Returning instance by path
07/26/2012 17:04:23.605 [Information] HPWmiSys.DLL!CHPWinComputerSystemList::RebuildList| GetApplianceData(): .ini file does not exist at the expected location
07/26/2012 17:04:23.605 [Information] HPWmiSys.DLL!CHPWinComputerSystemProv::GetInstanceByPath| Returning instance by path

 I've just reinstalled the WBEM provider. Seems like an .ini file is missing somewhere. Without help from HP i can't do anything.

[budykiller03]

I've try a new installation / Discovery for a Windows 2008R2 Core server.

- HP Providers installed, and i can see health status.

- Subscription works for this target ! So I will try to reboot the target machine TLSS001, maybe there's something wrong i can't explain.

- New event (same test with the N°2 NIC Card enabling / Disabling) doesn't show in HP SIM. I can see in NT event viewer the WBEM event but it is not reported to my HP SIM Server.

As a summary, for now we have :

- a target machine from which no subscription can be made -> will see on Monday after sunday reboot.

- a new target machine with subscription to WBEM events made but not reported to HP SIM

- can't reinstall agent with HP SIM page -> i've set "UAC" to off, i will see after next reboot if it works.

We ended up close to my first post. Events are not reported to HP SIM whatever the Windows target it cames from.

[budykiller03]

Something is not clear to me : on the target machine, is HP SMH mandatory to make all this work or can i only install HP WBEM Providers ?

[shocko]

You don't need the SMH for anything other than displaying all information collected by the providers. It's the providers that talk to the hardware and write events to the event logs as well as detecting faults etc.

[budykiller03]

Thank you for this clarification.

I still can't get it working.

- I can access root/HPQ namespace from other computers (with the admin account). I still dunno why i can't receive any event nor subscribe to them.

- I've just installed version 6.3 instead of 6.2, nothing changed.

- check logs : nothing special

- ran again Wireshark : i can see transactions going between both servers.

About packets : the only difference between the target i can subscribe to WBEM event, and the other one that don't work is :

- after Kerberos exchange, with RPC calls, the working target has multiple "IWbemCallStatus v0", rather the failing target has multiple "IWbemWCOSmartEnum v0"

[EDIT]

HPWmiSys.DLL!CHPIMLAccess::RebuildList| GetApplianceData(): .ini file does not exist at the expected location

 Regarding the above error, would someone be kind and check for any ".ini" files in his "HPWBEM" directory ? (on a windows target machine, with WBEM providers installed). I've checked mine, i don't have any INI file.

Thanks.

[shocko]

The account used to create the subscription must be an admin on the target server. Kerberos is used so make sure the times are in sync with the CMS. Use the mxwbemsub command to see if subscriptions are in place.

[budykiller03]

mxwbemsub in the only tool i use because through the CMS it never works.

Otherwise :

- mxwbemsub -l -n tlss002 returns "subscription OK"

- mxwbemsub -l -n tlss001 returns "no match found"

- mxwbemsub -a -n tlss001 returns :

192.168.0.1 FAILED to create indication subscription.

Cause: Unable to create a socket connection on the managed system

Recommended Action: Managed system must be listening on socket 5989 (WBEM) or 2381 (System Management Homepage).  If the managed system is running Windows, then ensure System Management Homepage is running.  Otherwise ensure WBEM is accessible.

 - no active firewall, and  a sniffer sees traffic between both machines.

[shocko]

The CMS should just be calling this executable so ensure that the permissions on the directory it resides in have not been changed and the account the SIM service is running under has full access to it. On the system where the subscription creation failed run this and post the output:

• netstat -anob

We need to see what's listening on that port. Also, double check that webem is listed as a detected protocol for that system in SIM and that admin credentials are listed in the credentials list for that system.

[budykiller03]

1) CMS rights

i'm using the global admin username to be sure it has all the rights (CMS Sign-IN + WMI/WBEM credentials, etc).

Like i said a few post above, credentials are working fine, because i got Health Status from my server and i can see "WBEM" in the property of the target, in the "protocol" line. (it even added HTTPS & SMH when i installed it yesterday).

i've also modified a month ago all the NTFS security to the BIN directory of HP SIM as soon as i got errors from the CMS when lunching subscription, and not from the command-line. It doesn't change anything however.

2) netstat command

See private message. If someone else would like to help me, i can give you the link.

Thanks

[ICS]

My understanding is the ESXi 5 offline bundle for Windows 6.3 is no longer supported.  sucks you have to upgrade to SIM 7.1

[budykiller03]

ESXi is fully functionnal. All hosts using ESX are monitored perfectly.

Only real MS hosts are buggy.

Anyway... i upgraded to HP SIM 7.1. Nothing changed. Same errors.

Can't create subscription for 192.168.0.1

No events received from 192.168.0.2 despite subscription...

SNMP doesn't work too. No versions of installer are compatible.

I've done all i could by my own and the help of a few (thanks shoko especially).

I'm really mad against HP