Safely shutdown SimpliVity cluster with internal vCenter Server Appliance

[jfc]

Hello,

What is the procedure to safely shutdown a 4 nodes SImpliVity cluster with vCSA VM hosted by this cluster?

Thanks !

[gustenar · Accepted solution]

Hello @jfc 

Normally you would shutdown all vms (or evacuate them) and shutdown the Omnistack controller  before shutting down the host. The recommended method to shutdown the Omnistack controller is to connect via putty (SSH) and run the command "svt-shudown-safe" and wait until it's down. 

Now, this command won't work when you you power off the VCSA VM so you need an alternate method. I would suggest doing the following: 

1. Work first on the hosts that don't have the VCSA running on them. 

2. Shutdown all VMs. 

3. Connect via putty to the Omnistack controller and run the command "svt-shutdown-safe". Wait until you see it turned off in vcenter. 

4. Put the node in maintenance mode and shut it down. 

For the host that has VCSA running on it:

1. Shutdown all VMs except OmniStack controller. 

2. Connect to the Omnistack controller via putty (SSH). Since VCSA will be down, you won't be able to use Vcenter credentials. Instead, use your svtcli account to log in. 

3. Once logged in, run these commands: 

-- sudo su

-- source /var/tmp/build/bin/appsetup

-- stop svtfs (wait until you get the prompt back and says "stop/waiting")

-- shutdown -hy now

This is going to shutdown the OVC gracefully. 

Continue and put this host in maintenance mode and shut it down. 

NOTE: The above procedure assumes you are running software version 3.7.3 or later. If you are on an older version you will need to contact support for assistance. 

Hope it helps, 

Gus

[fahlis]

Hi!

@gustenar 

Is this still a valid method to properly shutdown a SimpliVity cluster?

I need to shutdown a customers 2-node cluster for maintenace next week.

Running on OmniStack 4.0.1

Br / Tony

[gustenar]

Hello @fahlis 

The only command that changes is -- stop svtfs (wait until you get the prompt back and says "stop/waiting")

Use instead "systemctl stop svtfs@0" in version 4.0.1

[fahlis]

@gustenar
Many thanks.
Then I feel confident.
Is there perhaps a white paper available for the procedure?

Br / Tony

[gustenar]

@fahlis 

Check our video library with the procedure:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sf000045407en_us

[fahlis]

@gustenar
Great, even better
Br / Tony

[fahlis]

@gustenar
I had some issues with this.
It worked just fine until after the step.
"systemctl stop svtfs@0"
At this point the OVC stalled and nothing happened for approx 30min.
I then decided to power it off.
Set the host in maintenance mode and shut down.
Crossed my fingers and booted up after maintenance window and found all VMs on that node as "inaccessible"
Exit maintenance mode.
The OVC could not start.
Enter maintenance mode again.
One more reboot and lucky for me it worked.
I don't know what the cause of OVC stalling was.
Any hints?

Br /Tony

[Randolfini]

@fahlis 

I replied to this some time ago but it seems my post never made it. So I'll try again, albeit more condensed than my original.

I had the same problem when I first performed maintenance on an inherited SimpliVity Cluster, specifically one I upgrade from one node to two nodes. I was doing a firmware upgrade and had to fail-over to one node. The arbiter and VCSA are running inside the cluster, which isn't best practice, but there you go. When I failed over to one node I lost all storage as the remaining OVC split-brained, all the VMs just stopped including vCenter, OVC, etc. My problem was maintenance of one and yours is shutdown of both of the nodes but similar problems happened.

The correct process is this:

1. Before you start, you'll want to make sure you have https access to each vmware node, vmware root account for each node, ssh access and account to OVC
2. SSH to an OVC in the cluster and check that each VM is in storage HA: svt-vm-show --violations​
3. Storage vMotion the Arbiter to local storage and compute of a node that will be shutdown last
4. Storage vMotion the VCSA to local storage and compute of the same node as 1 that will be shutdown last
5. On each of the nodes gracefully shutdown all VMs except the Arbiter, VCSA, and OVCs. Clearly do this in proper order for your environment, such as domain controllers last, etc.
6. Proceed with safe shutdown of the OVC of each node (Using SSH, one at a time), they should gracefully shutdown: svt-shutdown-safe​
7. Hosts should now successfully enter maintenance mode.
8. Shut down the VMware hosts until you get to the last one running Arbiter and VCSA
9. Sign-in to the HTTPS management page of this VMware host
10. The OVC should be shutdown, now gracefully shutdown the Arbiter and VCSA. You will lose VCSA oversight as it shuts down.
11. Shut down the host with the management page you opened earlier

The power on process is in reverse:

1. Power on each host
2. When fully up, power on VCSA and Arbiter, when fully up proceed
3. Ensure you can sign-in to VCSA without issue, if so proceed
4. Power on OVC on each host
5. SSH to each OVC and monitor the startup of SVTFS. Once it has a post-start process everything should be nearly running: watch status svtfs​
   Every 2.0s: status svtfs
   svtfs (0) start/post-start, process 3499 post-start process 5002
6. Staying on OVC check the federation status and should be connected: svt-federation-show​
7. Check to ensure the VMs are in HA then start powering them on: svt-vm-show --violations​   . If not let them sychronize then power on
8. Monitor servers and they should boot up ok

[fahlis]

@Randolfini
Thanks for your reply and guide. I will progress in that order next time. You really should take the Arbiter out of the Simplivity cluster. Having it within the Simplivity cluster is "Shoot yourself in the foot" as I'm sure you are aware of.