Event "Cannot login --hostip@OVC IP" occurs every 10 minutes

[guan8]

Hi,

We are running a 2+1 Simplivity Federation. We are getting a weird error in the event log of the ESXi host every 10 minutes. It only occurs on one host, the one have in a remote location for backups. It looks like something is trying to login to the ESXi host with username "--hostip@10.10.100.163", or maybe just "--hostip", but I'm not sure why the OVC IP is mentioned in a login attempt to the ESXi host.

10.10.100.63 is the IP address of the ESXi host.

10.10.100.163 is the IP address of the OVC on that ESXi host.

The event in full says:

Date Time:10/03/2020, 21:50:25

User:--hostip

Type:Error

Target: 10.10.100.63

Description:

 10/03/2020, 21:50:25Cannot login --hostip@10.10.100.163

Event Type Description:

A user attempted to log in with an unknown or invalid username

Possible Causes:

• The username is unknown to the system Action: If you are using Active Directory, check the health of the domain controller
• The user provided an invalid password Action: Supply the correct password

Related events:

There are no related events.

I tried googling it but didn't find anything. Thanks in advance.

/Gustav

[dhooley]

Hi @guan8,

Thanks for using the forum.

The most likely cause of this would be an issue between the stored OVC authentication token and the actual authentication information for the ESXi.

To resolve this, you will need to run the below command on the OVC in question (SSH to the Mgmt. IP):

dsv-digitalvault-init --hmsuser "administrator@vsphere.local" --hostip <ESXi-IP>  --hostuser root

Note: You will then be prompted for the vCenter admin password & the ESXi root password.

If the issue persists after this or you are not comfortable with the above, I would suggest opening a support ticket.

Reagrds,

David

I am a HPE SimpliVity employee.

[gustenar]

Hello @guan8 

Just to add to what @dhooley mentioned, I would also recommend making sure ESX root account is not blocked by now. 

One easy test is to try opening a putty session to it and see if you are able to log in. If it appears to be blocked it's better to have it reset before attempting to update the credentials. 

Also, to be able to run the command suggested in previous reply you need to obtain root access to the OVC. To do this, type the following two commands:

sudo su

source /var/tmp/build/bin/appsetup

[guan8]

Thank you for your quick reply.

I will try this tomorrow and get back to you.

/Gustav

[guan8]

The issue still persists, unfortunately.

This is the output from the command:

root@omnicube-ip100-163:/home/administrator@vsphere# dsv-digitalvault-init --hmsuser "administrator@vsphere.local" --hostip 10.10.100.63 --hostuser root
2020-03-12 19:22:00Z Updating the postgres user mgmt_usr with new password
2020-03-12 19:22:00Z Updating the postgres user svtaggregator with new password
2020-03-12 19:22:00Z Initializing Digital Vault with postgres user mgmt_usr svtaggregator

I actually remember that I opened a case with HPE way back in 2019 regarding something completely different, and the support agent did something similar. Here's the partial output from the commands back in 2019:

administrator@vsphere@omnicube-ip100-163:~$ sudo su
root@omnicube-ip100-163:/home/administrator@vsphere# source /var/tmp/build/bin/appsetup
root@omnicube-ip100-163:/home/administrator@vsphere# stop svt-controlservices && stop svt-hyperproxy-server && stop svt-rest ; start svt-hyperproxy-server ; start svt-controlservices ; start svt-rest
stop: Unknown job: svt-controlservices
start: Job is already running: svt-hyperproxy-server (0)
start: Unknown job: svt-controlservices
start: Job is already running: svt-rest
root@omnicube-ip100-163:/home/administrator@vsphere# dsv-digitalvault-init --hmsuser administrator@vsphere.local --hostip 10.10.100.63 --hostuser root
2019-10-19 18:39:13Z Updating the postgres user mgmt_usr with new password
2019-10-19 18:39:13Z Updating the postgres user svtaggregator with new password
2019-10-19 18:39:13Z Initializing Digital Vault with postgres user mgmt_usr svtaggregator
root@omnicube-ip100-163:/home/administrator@vsphere# dsv-update-vcenter --username administrator@vsphere.local --server 10.0.162.199

We are planning on upgrading Omnistack from 3.7.7 to 3.7.9 this weekend, but I'm guessing we should resolve this issue first, to avoid any misconfiguration during the upgrade?

I guess we should open a case with HPE. It's just that every time I open a case, I get the feeling that I'm talking to someone who's not very computer savvy, and is simply following very exact instructions from their third line support, without the possibility of explaining what the commands they're typing into our environment does. It's quite frustrating and a little worrying.

For example, as you can see in the output above, the support agent tried to stop some services and start them again, but failed to do so, and then that suddenly wasn't important anymore, or the support agent didn't really understand what happened, and simply continued.

/Gustav

[Bendilock]

EDIT: erf... wrong account! I'm guan8.

I opened a case and talked to a support agent who tried a few basic commands like:

svt-federation-show

dsv-digitalvault-init --hmsuser "administrator@vsphere.local" --hostip 10.10.100.63 --hostuser root

dsv-event-mgr testcert

and checking the last few minutes of $SVTLOG.

He then concluded this is definitely a vmware issue and not Simplivity related and opened a case with vmware for me.

/Gustav

[Bendilock]

I just checked the eventmgr.log on OVC 10.10.100.163 and found this, which correlates very well with the error in the GUI. Anyone have any idea how to correct this?

cat /var/svtfs/0/log/eventmgr.log

2020-03-13 13:51:19,018Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:01:20,030Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:01:20,218Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:11:21,184Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:11:21,367Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:21:22,250Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:21:22,436Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:31:23,507Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:31:23,686Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:41:24,747Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:41:24,927Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now
2020-03-13 14:51:26,008Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 14:51:26,189Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now

/Gustav

[guan8 · Accepted solution]

The issue has now been resolved!

The solution was issuing this command: dsv-digitalvault-init administrator@vsphere.local 10.10.100.63 root on the OVC in question.

The issue was introduced in October 2019 when an HPE support agent issued the command dsv-digitalvault-init --hmsuser "administrator@vsphere.local" --hostip 10.10.100.63 --hostuser root which apparently is not the correct syntax for the version of Omnistack (3.7.7) we are running.

The error in vCenter is gone, and now correctly displays the following every 10 minutes:

Date Time:13/03/2020, 16:50:04

User:root

Type:Information

Target: 10.10.100.63

Description:

 13/03/2020, 16:50:04User root@10.10.100.163 logged in as Java/1.8.0_60-internal

Related events:

There are no related events.

The warning in /var/svtfs/0/log/eventmgr.log still persists, and is shown in all OVC's:
2020-03-13 18:06:09,493Z INFO Thread-4 [c.s.e.OmniCubeManager] run(OmniCubeManager.java:91) - OmniCubeManager refreshing...
2020-03-13 18:06:09,685Z WARN Thread-4 [c.s.e.OmniCubeManager] checkHostSettings(OmniCubeVMWareManager.java:117) - Found hostSettings not set correctly, attempting to correct now

This might be a separate issue though, so I am setting this to resolved.

/Gustav