OVC reporting "missing credentials" after Dell OmniCube CN-2200 Accelerator Card chang

[TomiKarlsson]

Hi all. We just had to replace the Accelerator Card on a OmniCube CN-2200 node. Cluster and Federation shows ok, but in the $SVTLOG it shows:

2019-08-14T12:04:14.461Z INFO 0x7f1ab19b6700 [4089:48997976] [cfgdb.task.taskmanagerservice] taskmanagerservice.cpp:1621 Processing request task_request[follows_span_id = , id = 0759d3b5-38c3-46b5-9fcc-25b77ccb3d90, owner = 422c2014-89f8-c393-a4f9-d68499fa6764, parent_span_id = , seq_num = 0, span_id = 9e5220b2b0e50828, task_async_parent = 00000000-0000-0000-0000-000000000000, task_cancel_restart = 1, task_destination = 422c2014-89f8-c393-a4f9-d68499fa6764, task_feature_level = 1222, task_invisible = 1, task_parent = 00000000-0000-0000-0000-000000000000, task_priority = 2000, task_receive_time = 1565782212, task_stunned = 0, task_type = com.simplivity.task.node.refreshstorageclientaddrs, trace_id = 291bc0a35ef85f23280ae99eaa52f347]
2019-08-14T12:04:14.461Z ERROR 0x7f1ab19b6700 [4089:48997976] [control.task.node.refreshstorageclientaddrs] refreshstorageclientaddrs.cpp:82 Error failed to get address info:DigitalVaultServiceException:Failed to load records from digital vault with error: InvalidKeyException:Error finalising cipher data: pad block corrupted[17]
2019-08-14T12:04:14.461Z INFO 0x7f1ab19b6700 [4089:48997976] [cfgdb.task.taskmanagerservice] taskmanagerservice.cpp:1661 Task 0759d3b5-38c3-46b5-9fcc-25b77ccb3d90 Response:
2019-08-14T12:04:14.461Z INFO 0x7f1ab19b6700 [4089:48997976] [cfgdb.task.taskmanagerservice] taskmanagerservice.cpp:1691 Task Request task_request[follows_span_id = , id = 0759d3b5-38c3-46b5-9fcc-25b77ccb3d90, owner = 422c2014-89f8-c393-a4f9-d68499fa6764, parent_span_id = , seq_num = 0, span_id = 9e5220b2b0e50828, task_async_parent = 00000000-0000-0000-0000-000000000000, task_cancel_restart = 1, task_destination = 422c2014-89f8-c393-a4f9-d68499fa6764, task_feature_level = 1222, task_invisible = 1, task_parent = 00000000-0000-0000-0000-000000000000, task_priority = 2000, task_receive_time = 1565782212, task_stunned = 0, task_type = com.simplivity.task.node.refreshstorageclientaddrs, trace_id = 291bc0a35ef85f23280ae99eaa52f347]: Suspended for 00:01:00 (Result::checkAgainIn)
2019-08-14T12:04:15.234Z ERROR 0x7f1bb06d4700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:18.240Z ERROR 0x7f1bb03d1700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:21.295Z ERROR 0x7f1bb06d4700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:24.351Z ERROR 0x7f1bb03d1700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:27.406Z ERROR 0x7f1bb06d4700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:30.458Z ERROR 0x7f1bb06d4700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials
2019-08-14T12:04:32.919Z INFO 0x7f1a7f952700 [:] [control.hypervisor.hypervisorsyncservice] hypervisorsyncservice.cpp:207 HypervisorSyncService - executing full sync by request: 1
2019-08-14T12:04:32.919Z INFO 0x7f1a7f952700 [:] [control.hypervisor.hypervisorsyncservice] hypervisorsyncservice.cpp:830 SyncNodeWithVCenter processing started ...
2019-08-14T12:04:32.931Z ERROR 0x7f1a7f952700 [:] [control.hypervisor.hypervisorsyncservice] hypervisorsyncservice.cpp:1080 Error Failed to fetch failuredomain info from vcenter for local node with identity 422c2014-89f8-c393-a4f9-d68499fa6764 : DigitalVaultServiceException:Failed to load records from digital vault with error: InvalidKeyException:Error finalising cipher data: pad block corrupted
2019-08-14T12:04:32.931Z INFO 0x7f1a7f952700 [:] [control.hypervisor.hypervisorsyncservice] hypervisorsyncservice.cpp:856 SyncNodeWithVCenter processing postponed for 30 seconds due to pollHMS error ...
2019-08-14T12:04:32.931Z INFO 0x7f1a7f952700 [:] [control.hypervisor.hypervisorsyncservice] hypervisorsyncservice.cpp:315 Scheduling node sync for: Wait pollHMS ready in 30 seconds
2019-08-14T12:04:32.931Z INFO 0x7f1bb8fe6700 [:] [control.scheduler.scheduler] schedulerimpl.cpp:614 Received request to run HypervisorSyncServiceEvent event queueNodeSync at 2019-Aug-14 12:05:02
2019-08-14T12:04:32.931Z INFO 0x7f1bb8fe6700 [:] [control.scheduler.scheduler] schedulerimpl.cpp:627 Waiting 00:00:30 to run HypervisorSyncServiceEvent event queueNodeSync
2019-08-14T12:04:33.522Z ERROR 0x7f1bb03d1700 [:] [control.svtcontrolhandler] federation.cpp:160 SvtControlHandler throws MissingCredentials

Any ideas? Anybody?

Best regards, Tomi

[gustenar]

It looks like you need to update the digitail vault (identity store if on an earlier version) to fix those errors. 

The command would be "dsv-digitalvault-init" or "dsv-identitystore-init" depending on what software version you are running. 

You can use the --help switch to know the require options but basically you need to include three parameters in the command : HMS username, Host IP, Host Username. For instance:

dsv-identitystore-init administrator@vsphere.local 111.222.333.444 root

The system will prompt you for the passwords.

Hope this helps. 

Gus

[TomiKarlsson · Accepted solution]

Thanks, the dsv-digital-vault-init had to be used with 3.7.8.

Providing the correct syntax here. Might be useful for others too

sudo su
source /var/tmp/build/bin/appsetup

root@omnicube-ip115-22:/home/administrator@vsphere# dsv-digitalvault-init --hmsuser administrator@vsphere.local --hostip 192.168.115.11 --hostuser root

This will delete any existing digital vault records and reinitiate new ones, are you sure you want to proceed.
Proceed? (y/n): y

Password for HMS user administrator@vsphere.local :
Password for OmniStack host user root@192.168.115.11 :
2019-08-14 12:53:40Z Updating the postgres user mgmt_usr with new password
2019-08-14 12:53:41Z Updating the postgres user svtaggregator with new password
2019-08-14 12:53:41Z Initializing Digital Vault with postgres user mgmt_usr svtaggregator
vCenter IP: 192.168.115.34
Successfully updated vCenter credential.
Successfully reinitialized digitlaVault.
root@omnicube-ip115-22:/home/administrator@vsphere#

[TomiKarlsson]

Also, had to stop and start svtfs on OVC node where card was changed.

/Tomi

[gustenar]

@TomiKarlsson 

Updating the digital vaults doesn't require reboot. Did you have to stop svtfs for any other reason?

During the replacement process of the TIA card there is a need to stop the OVC and shut it down to be able to put the node in maintenance mode, if that what you were referring to.