Safely shutdown SimpliVity cluster with internal vCenter Server Appliance

[jfc]

Hello,

What is the procedure to safely shutdown a 4 nodes SImpliVity cluster with vCSA VM hosted by this cluster?

Thanks !

[gustenar · Accepted solution]

Hello @jfc 

Normally you would shutdown all vms (or evacuate them) and shutdown the Omnistack controller  before shutting down the host. The recommended method to shutdown the Omnistack controller is to connect via putty (SSH) and run the command "svt-shudown-safe" and wait until it's down. 

Now, this command won't work when you you power off the VCSA VM so you need an alternate method. I would suggest doing the following: 

1. Work first on the hosts that don't have the VCSA running on them. 

2. Shutdown all VMs. 

3. Connect via putty to the Omnistack controller and run the command "svt-shutdown-safe". Wait until you see it turned off in vcenter. 

4. Put the node in maintenance mode and shut it down. 

For the host that has VCSA running on it:

1. Shutdown all VMs except OmniStack controller. 

2. Connect to the Omnistack controller via putty (SSH). Since VCSA will be down, you won't be able to use Vcenter credentials. Instead, use your svtcli account to log in. 

3. Once logged in, run these commands: 

-- sudo su

-- source /var/tmp/build/bin/appsetup

-- stop svtfs (wait until you get the prompt back and says "stop/waiting")

-- shutdown -hy now

This is going to shutdown the OVC gracefully. 

Continue and put this host in maintenance mode and shut it down. 

NOTE: The above procedure assumes you are running software version 3.7.3 or later. If you are on an older version you will need to contact support for assistance. 

Hope it helps, 

Gus

[fahlis]

Hi!

@gustenar 

Is this still a valid method to properly shutdown a SimpliVity cluster?

I need to shutdown a customers 2-node cluster for maintenace next week.

Running on OmniStack 4.0.1

Br / Tony

[gustenar]

Hello @fahlis 

The only command that changes is -- stop svtfs (wait until you get the prompt back and says "stop/waiting")

Use instead "systemctl stop svtfs@0" in version 4.0.1

[fahlis]

@gustenar
Many thanks.
Then I feel confident.
Is there perhaps a white paper available for the procedure?

Br / Tony

[gustenar]

@fahlis 

Check our video library with the procedure:

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=sf000045407en_us

[fahlis]

@gustenar
Great, even better
Br / Tony

[fahlis]

@gustenar
I had some issues with this.
It worked just fine until after the step.
"systemctl stop svtfs@0"
At this point the OVC stalled and nothing happened for approx 30min.
I then decided to power it off.
Set the host in maintenance mode and shut down.
Crossed my fingers and booted up after maintenance window and found all VMs on that node as "inaccessible"
Exit maintenance mode.
The OVC could not start.
Enter maintenance mode again.
One more reboot and lucky for me it worked.
I don't know what the cause of OVC stalling was.
Any hints?

Br /Tony

[Randolfini]

@fahlis 

I replied to this some time ago but it seems my post never made it. So I'll try again, albeit more condensed than my original.

I had the same problem when I first performed maintenance on an inherited SimpliVity Cluster, specifically one I upgrade from one node to two nodes. I was doing a firmware upgrade and had to fail-over to one node. The arbiter and VCSA are running inside the cluster, which isn't best practice, but there you go. When I failed over to one node I lost all storage as the remaining OVC split-brained, all the VMs just stopped including vCenter, OVC, etc. My problem was maintenance of one and yours is shutdown of both of the nodes but similar problems happened.

The correct process is this:

1. Before you start, you'll want to make sure you have https access to each vmware node, vmware root account for each node, ssh access and account to OVC
2. SSH to an OVC in the cluster and check that each VM is in storage HA: svt-vm-show --violations​
3. Storage vMotion the Arbiter to local storage and compute of a node that will be shutdown last
4. Storage vMotion the VCSA to local storage and compute of the same node as 1 that will be shutdown last
5. On each of the nodes gracefully shutdown all VMs except the Arbiter, VCSA, and OVCs. Clearly do this in proper order for your environment, such as domain controllers last, etc.
6. Proceed with safe shutdown of the OVC of each node (Using SSH, one at a time), they should gracefully shutdown: svt-shutdown-safe​
7. Hosts should now successfully enter maintenance mode.
8. Shut down the VMware hosts until you get to the last one running Arbiter and VCSA
9. Sign-in to the HTTPS management page of this VMware host
10. The OVC should be shutdown, now gracefully shutdown the Arbiter and VCSA. You will lose VCSA oversight as it shuts down.
11. Shut down the host with the management page you opened earlier

The power on process is in reverse:

1. Power on each host
2. When fully up, power on VCSA and Arbiter, when fully up proceed
3. Ensure you can sign-in to VCSA without issue, if so proceed
4. Power on OVC on each host
5. SSH to each OVC and monitor the startup of SVTFS. Once it has a post-start process everything should be nearly running: watch status svtfs​
   Every 2.0s: status svtfs
   svtfs (0) start/post-start, process 3499 post-start process 5002
6. Staying on OVC check the federation status and should be connected: svt-federation-show​
7. Check to ensure the VMs are in HA then start powering them on: svt-vm-show --violations​   . If not let them sychronize then power on
8. Monitor servers and they should boot up ok

[fahlis]

@Randolfini
Thanks for your reply and guide. I will progress in that order next time. You really should take the Arbiter out of the Simplivity cluster. Having it within the Simplivity cluster is "Shoot yourself in the foot" as I'm sure you are aware of.

[fahlis]

Hi!

I have just done a shutdown/ startup for maintenance for a customer. This is the steps I followed successfully.

Shutdown

1. Work first on the nodes that don't have the vCSA running.
2. Shutdown all VMs. 
3. Connect via putty to the OVC and run the command "svt-shutdown-safe". Wait until you see it turned off in vCenter. 
4. Put the node in maintenance mode and shut it down.
5. For the host that has vCSA running on it:

           Shutdown all VMs except the OVC. 

           Connect to the OVC via putty (SSH). Since vCSA will be down, you won't be able to use vCenter credentials.                         Instead, use your svtcli account to log in. 

          Once logged in, run these commands:

          svt-session-start

          Logon with the vCenter SSO account

         sudo su

         source /var/tmp/build/bin/appsetup

         systemctl stop svtfs@0

         shutdown -h now

        This is going to shutdown the OVC gracefully. 

        Continue and put this host in maintenance mode and shut it down

Startup

1. Power on all nodes
2. Logon to the nodes  WebUI
3. Power on all OVCs, wait until they are fully up and running, you can determine that by seeing that all the storage is availabe and VMs are not seen as "unavailable"
4. Power on vCSA
5. Power on VMs, DCs first and so on..
6. Logon to one of the OVCs and verify SVT Federation and VM HA

          svt-federation-show

          svt-vm-show>

         Done

        The previous shutdown issue I had was due to the wrong command given earlier in the thread
        "shutdown -hy now"

By a hunch instead I used "shutdown -h now" and it worked

[jlangmead]

Hi @fahlis 

Thanks for the info - that's useful stuff.

Just a quick query on your process - you say to shutdown all the VMs except the OVC and then to log in using the SVTCLI user as vCenter is offline. Having run the svt-session-start command you then say to login using the vCenter SSO account - should that be successful with the vCenter offline? Does the svt-session-start command allow this to succeed without needing to contact the VCSA for authentiucation?

kind regards

Jon

[fahlis]

Hi @jlangmead 

That's correct, the svt-session-start command will allow it to proceed. I guess the OVC...once you have logged in has the SSO credentials cahced.

[Stefano Colombo]

hello all

We're in a similar situation

After a complete power off for maintenance we have to power on a 2-node simplivity cluster, however the arbiter which was on a separate infrastructure cannot be brought online

My question is, can we simply boot the two OVC sequentially and wait for the federation be back online or the arbiter is mandatory for the power-on ?

thanks

[csa_autostore]

Having recently install a 2 node cluster on 4.1.0 these are the steps that worked for me and have been collated from HP support documentation. Its mostly what others have written but with some amendments to fit the HP guides.

These are the steps we use when we have vCenter (Windows server or VCSA) within the cluster.

--Shutdown--

*Log into vCenter

*vMotion all VM's over to one node (This is just my personal preference, we don't have very busy clusters and they are beefy enough to run everything on one node. You don't have to if you don't want/can't)

-If moving all VM's to one node. Wait for storage HA/sync messages to disappear.

*Shutdown all virtual machines except OVC's and VCSA

-Right click on each VM and select Power→Shutdown Guest OS

*Shutdown OVC on the Host that has no running VM's

-On the host you want to shutdown the OVC on. Through the vCenter console, right-click and select All HPE SimpliVity Actions→ Shut Down VirtualController. Click Yes to confirm the shutdown process
-Alternatively, you can ssh to the OVC directly, logging in with your vCenter SSO credentials.

sudo su
source /var/tmp/build/bin/appsetup
svt-shutdown-safe

*Shutdown vCenter

-Log into the vCenter Server Appliance Management Interface (https://FQDN-of-VCSA:5480 Log in as root or vCenter SSO user)

Click Summary.
From the top menu pane, click the Actions drop-down menu.
Click Shutdown to power off the virtual machine.
In the confirmation dialog box, click Yes to confirm the operation.

-For windows vCenter, you can shut this down like any other VM from within the vCenter console.

If you really want to be **bleep** about it.
log into the VM
Run CMD as admin
Cd 'C:\Program Files\VMware\vCenter Server\bin'
service-control --stop --all
Shutdown windows as normal

*Shutdown last remaining OVC

-ssh to the last remaining OVC directly logging in with your SVTCLI user

sudo su
source /var/tmp/build/bin/appsetup
svt-shutdown-safe
svt-shutdown-force (if this doesn't work you can run 'svt-shutdown-force --emergency')

*Place hosts in maintenance mode

-Log into each ESXI host via web console using root user
Right click on each host and select 'Enter maintenance mode'

*Shutdown Hosts

-Right click on each host and select 'shutdown'

--Startup--

*Power on all nodes

-Log into ILO of ESXI nodes and hit the power button

*Take nodes out of maintenance mode + startup OVC's + vCenter

-Log into each ESXI host via web console using root user
-Right click on each host and select 'Exit maintenance mode'
-Right click OVC and select power on
-One they are booted, right click VCSA and power it on.

*Check your cluster to ensure its healthy

-Logon to one of the OVCs using your vCenter SSO user and verify SVT Federation and VM HA

svt-federation-show (look for green text saying 'Alive' and 'connected'
svt-vm-show (look for green text saying 'yes')

*Power on All remaining VM's

-Log into vCenter and power on all VM's. (vMotion them back to original hosts now if you want before switching them on)

Done

There was another user on this thread that had the arbiter hosted within the cluster. I don't think HP support this kind of deployment and as such, might complain about it if you have any issues with it and try to get support through them. The Arbiter is meant to be a voting node on separate infra from the SimpliVity cluster.

Another user asked if the arbiter needed to be online before you could start up a cluster and the answer is no. You can bring a cluster online with the Arbiter offline. If the infra its on is dead. I believe there is a way to register the federation with a new Arbiter.