Omnistack 3.7.8 deployment failure step 29/33 Legacy node omnicube cn2200

[simonsimon1129]

Hello everyone
Machine: Omnicure CN-2200
OVC: 3.7.8
Java: JRE_ 1.8.0_221
Any tips would be a big help thanks.

Here's the deployment logs:

Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Thrift.Transport.TStreamTransport.Write(Byte[] buf, Int32 off, Int32 len)
at Thrift.Transport.TBufferedTransport.Flush()
at com.simplivity.deploy.Orchestrator.OrchestratorService.Client.send_terminate()
at Simplivity.DM.CAL.Impl.ServiceWrapper.KillOrchestrator(Int32 port)

Here's the orchestrator logs:

2021-02-09 12:24:38,129Z ERROR OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] error(DeployLogger.java:98) - step 29 of 33 (46.27%) - Failed to make a required system configuration change - Could not find the drive to create partitions on for the databases Could not find the drive to create partitions on for the databases
2021-02-09 12:24:38,129Z ERROR OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] rememberLowestLogLevel(Orchestrator.java:235) - Setting lower log level
2021-02-09 12:24:38,157Z INFO main [c.s.d.o.Orchestrator] deployAndLog(Orchestrator.java:1116) - Deployed 0 OmniCubes
2021-02-09 12:24:38,158Z DEBUG main [c.s.d.o.Orchestrator] waitForThriftTerminationToBeServiced(Orchestrator.java:695) - waiting for thrift server thread
2021-02-09 12:24:38,242Z DEBUG Thread-1 [c.s.d.o.Orchestrator] terminate(OrchestratorServiceHandler.java:116) - terminate

[gustenar]

Hello @simonsimon1129 

The problem appears to be related to the logical volumes. Before factory resetting the node it is required to clear the logical volumes of both the rear and front disks. The RAID 1 volume for the rear disks must be recreated and we let Deployment Manager take care of creating the logical volumes for the front disks. Did you perform these operations before Factory Resetting the node?

[simonsimon1129]

Hello @gustenar ,
thank you for your comment.

We have deleted the logical volumes in the server, and recreated the raid 1. and leave the others with no config
we have factory reset it and it was successful,
But upon deployment is giving us the same error.
Anything we need to check further ?

[gustenar]

Do you have another node of same model and hardware configuration? You can try the procedure again (delete and recreate RAID 1 for rear disks), but this time also re-create the logical volumes for the front disks. There should be 3 logical volumes created and then try redeployment. The size of the logical volumes must be set correctly, reason why I asked to check another node. 

[simonsimon1129]

Hello @gustenar 
Sorry we dont have any machine with the same hardware config since this is a legacy server old one.

we'll try to recreate the logical drive configure recently. before I remove it. and test it again


Anymore tips for us to do ? .....

[simonsimon1129]

@gustenar 
Hello Gustenar we did the action plan Recreate the old config of logical drives. but it shows us the same error.
Anything more we can do for us to continue the redeployment ? ....

Thank you so much

[gustenar]

Can you look into the deployment logs for the error and post it here? Make sure to include at least 10 lines before the error and 10 after. 

[simonsimon1129]

HI @gustenar Here's the logs

ORCHESTRATOR LOGS:

2021-02-11 06:31:51,874Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.65%) - Configuring svtfs - VMW_CONFIGURING_SVTFS(10100) InstallTypes::Base::prepSvtfsConfig
2021-02-11 06:31:53,745Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.68%) - Generating SSH Keys - VMW_GENERATING_SSH_KEYS(10030) InstallTypes::ESX::LabESX5::hmsPrepSvtfsConfigAdditions
2021-02-11 06:31:54,556Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.68%) - Configuring networking - VMW_CONFIGURING_NETWORKING(10050) InstallTypes::Base::linuxInterfaceConfig
2021-02-11 06:31:59,121Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.68%) - Generating certificate - VMW_GENERATING_CERTIFICATE(10040) InstallTypes::Base::generateCertificate
2021-02-11 06:31:59,493Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.68%) - Configuring svtfs - VMW_CONFIGURING_SVTFS(10100) InstallTypes::Base::alignTIAFirmware
2021-02-11 06:32:16,559Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 24 of 33 (45.68%) - Deploy phase 1 complete - DFB_PHASE1_DONE(5010) InstallTypes::Control::endPhase
2021-02-11 06:32:16,559Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 25 of 33 (45.68%) - Initialize Certificates
2021-02-11 06:32:16,663Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:180) - step 25 of 33 (45.68%) - Successfully initializePKI
2021-02-11 06:32:16,664Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 26 of 33 (45.78%) - Send all IDs
2021-02-11 06:32:16,694Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 26 of 33 (45.78%) - ID sent to Virtual Controller identity store - localhost:svtcli
2021-02-11 06:32:16,696Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 26 of 33 (45.78%) - ID sent to Virtual Controller identity store - 172.18.0.209:root
2021-02-11 06:32:16,698Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 26 of 33 (45.78%) - ID sent to Virtual Controller identity store - omnicube.host
2021-02-11 06:32:16,699Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 26 of 33 (45.78%) - All IDs sent
2021-02-11 06:32:16,700Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 27 of 33 (45.78%) - Create Virtual Controller digital vault records
2021-02-11 06:32:17,834Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 27 of 33 (45.80%) - Virtual Controller digital vault records successfully created
2021-02-11 06:32:17,835Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 28 of 33 (45.80%) - Start registering the SimpliVity extension
2021-02-11 06:32:20,372Z DEBUG OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] debug(DeployLogger.java:224) - step 28 of 33 (45.85%) - Waiting 45000 ms for extension registration to complete
2021-02-11 06:33:03,525Z DEBUG sessionTickler [c.s.h.h.v.n.s.VMwareHostnameVerifier] verify(VMwareHostnameVerifier.java:84) - Verifying host '172.18.0.25'
2021-02-11 06:33:03,526Z DEBUG sessionTickler [c.s.h.h.v.n.s.VMwareHostnameVerifier] verify(VMwareHostnameVerifier.java:102) - Verifying host '172.18.0.25' - peer cert Subject DN 'OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US, CN=vcenterbcp.qc.gma.network'
2021-02-11 06:33:03,526Z DEBUG sessionTickler [c.s.h.h.v.n.s.VMwareHostnameVerifier] verify(VMwareHostnameVerifier.java:106) - Retrieved cert SubjectAltNames: [email@acme.com, 127.0.0.1, vcenterbcp.qc.gma.network]
2021-02-11 06:33:03,527Z DEBUG sessionTickler [c.s.h.h.v.n.s.VMwareHostnameVerifier] matchIpAddressByDnsName(VMwareHostnameVerifier.java:223) - Found address '172.18.0.25' via DNS lookup of subjectAltName 'vcenterbcp.qc.gma.network/172.18.0.25'
2021-02-11 06:33:03,527Z DEBUG sessionTickler [c.s.h.h.v.n.s.VMwareHostnameVerifier] matchIpAddressByDnsName(VMwareHostnameVerifier.java:227) - SubjectAltName 'vcenterbcp.qc.gma.network' matches host '172.18.0.25'
2021-02-11 06:33:03,530Z DEBUG sessionTickler [c.s.d.o.Orchestrator] run(SessionCloning.java:261) - Tickled Session: 93dce568-e59f-48a6-a82a-a600537bf53b
2021-02-11 06:33:05,378Z INFO OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] info(DeployLogger.java:214) - step 28 of 33 (46.27%) - Successfully registered the SimpliVity extension
2021-02-11 06:33:05,380Z ERROR OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] error(DeployLogger.java:98) - step 29 of 33 (46.27%) - Failed to (re)configure RAID set - Failed to apply the RAID config
Failed to apply the RAID config

2021-02-11 06:33:05,387Z ERROR OmniStack-172.18.0.209 [c.s.d.o.Orchestrator] rememberLowestLogLevel(Orchestrator.java:235) - Setting lower log level
2021-02-11 06:33:05,539Z DEBUG Thread-1 [c.s.d.o.Orchestrator] terminate(OrchestratorServiceHandler.java:116) - terminate
2021-02-11 06:33:06,104Z INFO main [c.s.d.o.Orchestrator] deployAndLog(Orchestrator.java:1116) - Deployed 0 OmniCubes
2021-02-11 06:33:06,104Z DEBUG main [c.s.d.o.Orchestrator] waitForThriftTerminationToBeServiced(Orchestrator.java:695) - waiting for thrift server thread
2021-02-11 06:33:06,104Z DEBUG main [c.s.d.o.Orchestrator] auditThriftInterfacePollingResultForThriftServiceTermination(Orchestrator.java:721) - thrift server terminated
2021-02-11 06:33:06,119Z INFO main [c.s.h.h.v.VMWareSessionImpl] disconnect(VMWareSessionImpl.java:553) - Disconnecting from 'https://172.18.0.25/sdk' ...
2021-02-11 06:33:06,123Z INFO main [c.s.h.h.v.VMWareSessionImpl] disconnect(VMWareSessionImpl.java:555) - Disconnected
2021-02-11 06:33:06,123Z ERROR main [c.s.d.o.Orchestrator] verifyPostDeploymentStatus(Orchestrator.java:1043) - Errors occurred, 1 of 1 OmniCubes failed to fully deploy
2021-02-11 06:33:06,124Z INFO main [c.s.d.o.Orchestrator] logJavaMemory(JavaMemory.java:66) - Maximum memory (bytes): 252706816 ~ 241MiB
2021-02-11 06:33:06,125Z INFO main [c.s.d.o.Orchestrator] logJavaMemory(JavaMemory.java:74) - Total memory (bytes): 252706816 ~ 241MiB
2021-02-11 06:33:06,125Z INFO main [c.s.d.o.Orchestrator] logJavaMemory(JavaMemory.java:78) - Allocated memory (bytes): 76752032 ~ 73MiB
2021-02-11 06:33:06,125Z INFO main [c.s.d.o.Orchestrator] logJavaMemory(JavaMemory.java:81) - Free memory (bytes): 175954784 ~ 167MiB
2021-02-11 06:33:06,125Z INFO main [c.s.d.o.Orchestrator] logJavaMemory(JavaMemory.java:85) - Calc free memory (bytes): 175954784 ~ 167MiB
2021-02-11 06:33:06,126Z INFO main [c.s.d.o.Orchestrator] logAndSystemExit(Orchestrator.java:264) - System.exit(3) : One or more OmniStack hosts failed to deploy.

SVT DEPLOYMENT LOGS:

2021-02-11 06:12:18,507 [1] INFO ConfigurationController [LogInfo] - Deployment Manager is trying to check if session is still active on vCenter.
2021-02-11 06:12:18,604 [1] INFO Simplivity.DM.Presenter.NavigationManagement.NavigationManager [LogInfo] - Navigating from ReadyToDeployView to MonitorDeploymentView
2021-02-11 06:12:18,619 [1] INFO Simplivity.DM.View.DeploymentViews.MonitorDeploymentView [LogInfo] - Initializing the Simplivity.DM.View.DeploymentViews.MonitorDeploymentView view.
2021-02-11 06:12:18,641 [21] INFO OrchestratorDeploymentProcessMgr [LogInfo] - Deploy submission process start
2021-02-11 06:12:18,645 [21] INFO OrchestratorDeploymentProcessMgr [LogInfo] - Initiating the deployment.
2021-02-11 06:12:18,650 [21] INFO Simplivity.DM.Threading.Impl.ThreadingManager [LogInfo] - Initiating the work item: 8a6e7537-fb85-4060-b659-7eb4a255a105, using task: 70689.
2021-02-11 06:12:18,652 [21] INFO OrchestratorDeploymentProcessMgr [LogInfo] - Running Deployments: 1
2021-02-11 06:12:18,652 [21] INFO OrchestratorDeploymentProcessMgr [LogInfo] - Deploy submission process end
2021-02-11 06:12:18,653 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Serializing deployment information provided by user to be passed to Orchestrator service.
2021-02-11 06:12:18,671 [25] INFO ConfigurationController [LogInfo] - Deploy XML file saved: C:\Users\DCO-TEST6\AppData\Local\DeploymentManager\DeployXml\GMA_BCP_21-02-11_14-12-18.xml
2021-02-11 06:12:18,684 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Deployment Manager succesfully started Orchestrator service for deployment.
2021-02-11 06:12:20,688 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Deployment Manager trying to connect to started Orchestrator service for deployment on port 9290.
2021-02-11 06:26:51,208 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Updating deployment status for J3202HR.
2021-02-11 06:26:58,672 [22] INFO ConfigurationController [LogInfo] - Deployment Manager is trying to check if session is still active on vCenter.
2021-02-11 06:27:01,211 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Updating deployment status for SERIAL NUMBER.(MULTIPLE ENTRY)
2021-02-11 06:27:32,577 [1] INFO Simplivity.DM.View.DeploymentViews.MonitorDeploymentView [LogInfo] - Initializing the Simplivity.DM.View.DeploymentViews.MonitorDeploymentView view.
2021-02-11 06:27:41,250 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Updating deployment status for SERIAL NUMBER.(MULTIPLE ENTRY)
2021-02-11 06:27:51,255 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Updating deployment status for SERIAL NUMBER.(MULTIPLE ENTRY)
2021-02-11 06:33:05,531 [25] INFO Simplivity.DM.DeploymentController.Impl.OrchestratorDeploymentStrategy [LogInfo] - Terminating Orchestrator for WorkItem id 8a6e7537-fb85-4060-b659-7eb4a255a105.
2021-02-11 06:33:08,556 [25] ERROR Simplivity.DM.CAL.Impl.ServiceWrapper [LogError] - Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Thrift.Transport.TStreamTransport.Write(Byte[] buf, Int32 off, Int32 len)
at Thrift.Transport.TBufferedTransport.Flush()
at com.simplivity.deploy.Orchestrator.OrchestratorService.Client.send_terminate()
at Simplivity.DM.CAL.Impl.ServiceWrapper.KillOrchestrator(Int32 port)

[simonsimon1129 · Accepted solution]

Hello everyone 
deployment is successful we have reset some rest services and then the deployment went successful
this services :

stop svt-hyperproxy-server && start svt-hyperproxy-server
stop svt-controlservices && start svt-controlservices
service svt-rest stop && service svt-rest start

Thanks everyone !