Nimble Windows Toolkit 7.0 - Group Registration Error

[klaughin]

I am trying to upgrade NWT 5.0 to 7.0 on a Windows Server 2012 R2 Hyper-V host which is part of a cluster, and part of this process requires that we register a group on the Nimble. I enter the MGMT IP address of the Nimble and use a poweruser on the Nimble for the connection. This upgrade was successful on one host, but is failing on the remaining two. The error I see mentions that I need to check the MGMT IP or the credentials, which I am sure are correct. 
In the Nimble setup logs I am seeing this error. 

09-14-2022 14:51:17.612 4508 6404 Information Logging at Level 'Information'
09-14-2022 14:51:17.612 4508 6404 Information ------------------------------------------------------
09-14-2022 14:51:17.613 4508 6404 Information Executing Cmdlet Set-NWTConfiguration
09-14-2022 14:51:17.613 4508 6404 Information ------------------------------------------------------
09-14-2022 14:51:17.613 4508 6404 Information Entered process record for SET-NWTConfiguration cmdlet
09-14-2022 14:51:17.626 4508 6404 Information Connecting to Nimble Group management IP=172.X.X.X as user=poweruser
09-14-2022 14:51:17.626 4508 6404 Information Cmdlet=Set-NWTConfiguration Application Name reported to group=172.X.X.X is Set-NWTConfiguration-7.0.0.31
09-14-2022 14:51:17.678 4508 6404 Warning Login failed with exception Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send.. Re-attempting login.
09-14-2022 14:51:18.685 4508 6404 Warning Login attempt:1
09-14-2022 14:51:18.685 4508 6404 Warning Caught exception during retry count 1 with message: Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send.
09-14-2022 14:51:19.696 4508 6404 Warning Login attempt:2
09-14-2022 14:51:19.700 4508 6404 Warning Caught exception during retry count 2 with message: Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send.
09-14-2022 14:51:20.701 4508 6404 Warning Login attempt:3
09-14-2022 14:51:20.705 4508 6404 Warning Caught exception during retry count 3 with message: Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send.
09-14-2022 14:51:20.706 4508 6404 Error ValidateUserCredentials failed against Nimble Group. Error=Invalid or unreachable Nimble group management=172.X.X.X specified with user name=poweruser. Please check they are valid and reachable through firewall. Run 'Set-NWTConfiguration' cmdlet to set valid Nimble group management and credentials for Windows user=domain\user. Error Message=Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send.
09-14-2022 14:51:20.713 4508 6404 Error Failed to Set NWT configuration ,Exception:Nimble.Powershell.PSException.NimblePSException: Specified user credentials cannot be verified. Please check the group management IP or the credentials provided as input. ---> Nimble.Powershell.PSException.NimblePSException: Invalid or unreachable Nimble group management=172.X.X.X specified with user name=poweruser. Please check they are valid and reachable through firewall. Run 'Set-NWTConfiguration' cmdlet to set valid Nimble group management and credentials for Windows user=domain\user. Error Message=Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send. ---> NimbleStorage.WS.GroupMgmtClientException: Resource:https://172.X.X.X:5392/v1/tokens, Error:The underlying connection was closed: An unexpected error occurred on a send. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at RestSharp.Http.WriteRequestBody(HttpWebRequest webRequest)
at RestSharp.Http.PostPutInternal(String method)
at RestSharp.Http.AsPost(String httpMethod)
at RestSharp.RestClient.DoExecuteAsPost(IHttp http, String method)
at RestSharp.RestClient.Execute(IRestRequest request, String httpMethod, Func`3 getResponse)
--- End of inner exception stack trace ---
at NimbleStorage.WS.GroupMgmtClient.HandleReturnResponse[T](RestRequest request, IRestResponse`1 response)
at NimbleStorage.WS.GroupMgmtClient.Execute[T](RestRequest request)
at NimbleStorage.WS.GroupMgmtClient.Post[T](String objectName, DataWrapper`1 clazz, T postobject)
at NimbleStorage.WS.V1.TokenObjectSet.CreateObject(Token thatobject)
at NimbleStorage.WS.GroupMgmtClient.SetupGroupMgmtClient(String groupAddr, String userName, SecureString password, String appName)
at NimbleStorage.WS.GroupMgmtClient.Login(String groupAddr, String userName, SecureString password, String appName, UInt16 portNumber)
at Nimble.Powershell.Cmdlets.NimPSCmdletBase.CreateGroupManagementClient(GroupMgmtProfileConfigInfo GroupUser)
--- End of inner exception stack trace ---
at Nimble.Powershell.Cmdlets.NimPSCmdletBase.CreateGroupManagementClient(GroupMgmtProfileConfigInfo GroupUser)
at Nimble.Powershell.Cmdlets.SetNWTConfigurationCmdlet.ValidateUserCredentials(GroupMgmtProfileConfigInfo ConfigInfo)
--- End of inner exception stack trace ---
at Nimble.Powershell.Cmdlets.SetNWTConfigurationCmdlet.ValidateUserCredentials(GroupMgmtProfileConfigInfo ConfigInfo)
at Nimble.Powershell.Cmdlets.SetNWTConfigurationCmdlet.CreateNWTConfigProfile(GroupMgmtProfileConfigInfo record)
at Nimble.Powershell.Cmdlets.SetNWTConfigurationCmdlet.ProcessRecord()

I noticed that, on the Nimble GUI under the user events, I am seeing login attempts from the windows server where the upgrade that worked, but I am not seeing events from the windows servers where login attempts failed. 

I ran a wireshark capture during a failed login and noticed that the Nimble was sending Server-RST packets but I cannot see why. 

Any help is appreciated 

[Mahesh202]

Hi klaughin

Could you please Check the network configuration of the array,?
Also, please try again using the management IP and check you are trying to use the discovery IP to set up instead of the array mgmt IP.

Regards
Mahesh.

[klaughin]

As I mentioned I am using the Management IP. I have also successfully registered the group on another NWT installation on the same network so I am confident in my setup of the Nimble HF20. 

[Mahesh202 · Accepted solution]

Hi klaughin

1)Check to see if you have the ^ character in their password

2) TLS v1 is disabled on the host
If u followed proper security processes, they may have disabled TLS v1 on the Windows host. This is a good thing, but what may not be realized is that PowerShell and .net will still attempt to use TLS v1 even though it is disabled. This means when attempting to validate the credentials on the array, the initial negotiation with the array will fail.
3) - A proxy is in use on the host.
If the user connects via a proxy server, PowerShell does not use this proxy by default. You may see the following in the logs:
Warning Login failed with exception Resource:https://tokens, Error: The remote server returned an error: (407) Proxy Authentication Required. Re-attempting login.

Proxy must be enabled in PowerShell
$wc = New-Object System.Net.WebClient
$wc.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredential
4)- Required Cipher Suite not enabled
Currently, the Nimble array supports two strong ciphers ( TLS_RSA OR TLS_DHE ) and must enable either one of them to get a connection from the host's side. Specifically, the array uses TLS_RSA_WITH_AES_256_CBC_SHA256 when using the TLS_RSA ciphers.
To get a list of enabled Cipher Suites, from PowerShell run: Get-TlsCipherSuite | Format-Table -Property Name
You can enable using this command: Enable-TlsCipherSuite -Name TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDHE is NOT SUPPORTED

Hope this helps.!!

Regards
Mahesh

[klaughin]

Thank you for this. The issue ended up being the TLS settings for .NET. 

I created these two registry keys on the server and NWT was able to register the group successfully. Must be an issue with TLS capabilities since Server 2012 R2 is an older OS. 

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

[Mahesh202]

Hi klaughin

I am glad that the issue has been resolved.

Regards
Mahesh.