External WBEM Requests fail due to empty provider-list

[LS_ger]

Hi Folks, after quite a lot of searching for the cause of my issue, I think I found something interesting and I'm not sure how to fix it.

The problem occours with the following combination:

DL380 Gen10 with VMware ESXi 6.5.0 14990892 (using the HPE-Image)
HPE SIM (C.07.06.00.00r76rc05, Hotfixes until Hotfix76_063, Jan 2020)

I already tried to "upgrade" the package via the Offline bundle:
https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX-ca7c5cf6636640c78a00036722#tab-historThe  which yields: No upgraded packages found, so the HPe-Stuff seems to be still recent.

If I query the Configuration (e.g. via Discovery in HPE) I get no hardware status via WBEM.
Looking at the system via shell, the providers list yields:

esxcli system wbem provider list
Name                Enabled  Loaded
------------------  -------  ------
.#vmw_smx-provider     true   false
sfcb_base              true    true
vmw_base               true    true
vmw_hdr                true    true
vmw_hhrcwrapper        true    true
vmw_iodmProvider       true    true
vmw_kmodule            true    true
vmw_omc                true    true
vmw_pci                true    true
vmw_smx-provider       true    true
vmw_vi                 true    true

The .#vmw -Entry looked strange, so I looked into /var/lib/sfcb/registration and indeed found:

vmw_smx-provider-providerRegister (1308 Bytes)
.#vmw_smx-provider-providerRegister (77881 Bytes)

Comparing that with an older DL380 Gen9 Host within the same environment, the only had a vmw_smx-provider-providerRegister with 77881 Bytes. As i was exaclty the same bytesize, I dediced to overwrite the contents while disabling wbem via

esxcli system wbem set --enable=false

Afrer I enabled it back, this seemed to fix my issue and I was happy (discovery in HPe SIM yielded results). Until ne next reboot ...
It seems like the vmw_smx-provider-providerRegister gets overwritten with the initial 1308 Byte sized Version on reboot. Is this a known Issue and how can I get around this ?

I saw: https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00098388en_us
But this refers to pre 3.5.0 Versions ans should be fixed (and I saw no crashes so far). But there is also the strange .#vmw_smx-provider-providerRegister listed inside. A simple /sbin/auto-backup.sh didn't help to persist the settings through reboot.

[ksram]

Hi @LS_ger ,

May we know if you have ttied the workarounds mentioned in the Article.

Couldnt find any other artilce though.

Thank you

[LS_ger]

Hi,

this din't change anything. It also seemed a bit strange to disable the providers I want for them to work.
I also can't manage to get the provider ".#vmx_smx-provider to status loaded true.

so it doesn't help

[ksram]

Hi LS_ger, Apologies for delayed reply. I understand now its version 3.5.0 and any chances of bringing it back to 3.4.5 and check? Also is there any other ESXI with previous version showing similar behavior? or with same configuration working fine? so that we can compare the versions and checl. Thank you

[LS_ger]

Hi, I was on vacation during the last few weeks.

I will check give the downgrade a shot.

I dont have a running Gen10 Server with 3.5.0, only Gen9 and they don't have that issue.
The Version of WBEM Providers shown in VMware is

650.03.14.00.3-4240417 on the old Gen9 Servers (without that Issue)
650.03.16.00.4-4240417 on the new Gen10 Servers.

[LS_ger]

The Issue stays the same when downgrading the vis smx-providers.

I tried

HPE_bootbank_smx-provider_650.03.14.00.3-4240417
HPE_bootbank_smx-provider_650.03.15.00.4-4240417

The Version from the Image was
HPE_bootbank_smx-provider_650.03.16.00.4-4240417

The Bytesize seems to be the same as well.
But i found this article, which mentions the ".#" Files: https://www.virtuallyghetto.com/2011/08/how-to-persist-configuration-changes-in.html

There is something wrong with the image. I have to prevent the vmw_smx-provider-providerRegister from getting overwritten.

[ksram]

Hi @LS_ger ,

Check if there any new Patches available from VMware and try to update the same.

And check. How many units has this issue ? and whats the impact of this issue?

Thank you

[LS_ger]

The Issue persists on all Gen10 Servers.

The package for die limited Providers: HPE_bootbank_smx-limited_650.03.16.00.1-4240417

however does not suffer that Issue:

esxcli system wbem provider list

Name                Enabled  Loaded
------------------  -------  ------
sfcb_base              true    true
vmw_base               true    true
vmw_hdr                true    true
vmw_hhrcwrapper        true    true
vmw_iodmProvider       true    true
vmw_kmodule            true    true
vmw_omc                true    true
vmw_pci                true    true
vmw_smx-limited        true    true
vmw_smx-provider       false   true
vmw_vi                 true    true

The small / default  vmw_smx-provider-providerRegister is this there, but is not used in this scanario:

ls -l /var/lib/sfcb/registration
total 100
-rw-rw-rw-    1 root     root           128 Sep  7 13:39 loaded
drwxr-xr-x    1 root     root           512 Sep  7 13:38 repository
-r--r--r--    1 root     root          3513 Nov  1  2019 sfcb_base-providerRegister
-r--r--r--    1 root     root          4307 Nov  1  2019 vmw_base-providerRegister
-r--r--r--    1 root     root           374 Nov  1  2019 vmw_hdr-providerRegister
-r--r--r--    1 root     root           897 Nov  1  2019 vmw_hhrcwrapper-providerRegister
-r--r--r--    1 root     root          4220 Nov  1  2019 vmw_iodmProvider-providerRegister
-r--r--r--    1 root     root           164 Nov  1  2019 vmw_kmodule-providerRegister
-r--r--r--    1 root     root         15650 Jun 21  2019 vmw_omc-providerRegister
-r--r--r--    1 root     root          1550 Nov  1  2019 vmw_pci-providerRegister
-r--r--r--    1 root     root         31636 Aug  9  2019 vmw_smx-limited-providerRegister
-rw-r--r--    1 root     root          1308 Sep  7 13:39 vmw_smx-provider-providerRegister
-r--r--r--    1 root     root          2898 Mar 15  2018 vmw_vi-providerRegister

-> Also, I can't find and .# File inside that directory.

As a result, only a subset of WBEM date is available, which is better then nothing.
I have beed digging through the change logs of all update released, but this issue is not mentioned (just memory leaks etc.). There is an Issue, which causes the correct vmw_smx-provider-providerRegister to be overwritten.