Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

[Major88]

Hello,

I am looking for best pratics (procedure) for correct shutdown of 3 nodes Simplivity cluster v.40.0.

Firstly i shutdown all VMs exept vCenter

I shutdown one of the virtual controller by the Simplivity menu in the vcenter plugin. - it's ok.
Maintenance mode -is ok too.

But when i shutdown 2nd virtual controller, the vcenter is became unresponsive. And at this moment i can't login to ssh session with vcenter autentification to use svt-shutdown-safe.

and i blocked it this state. If is there other solution for completly shutdown the cluster's node with hosted vcenter (vcsa)

[DaveOb]

This is happening because the nodes on which VC  is stored have both been powered off,

https://damianerangey.com/ for a good explanation of how vm copies are managed.

You should be able to identify  from the DRS rule the location of the VC copies and power off the node without any VC copy first then a node with one copy.

This should leave you with one OVC running and VC.

Put 2x nodes into maintence mode and shutdown.

Power off VC gracefully.

SSH to the last remaining  OVC and issue a svt-shutdown-force command,

ssh to esxi or connect directly and shutdown,

Alternately you can just shutdown all VMs including VC and ssh to all 3 OVC and use the svt-shutdown-safe command and shutdown the ESXi via connecting directly.

[Major88]

i can connect to ssh, but without vcenter i can only login as svtcli user, and svtcli user can't use the svt commands:

i've this message

---

ERROR [47]: Missing session credentials.

[BenH10 · Accepted solution]

Try a --emergency on the end of the command. Some commands have that built in to ignore the session credentials requirement. I don't know what version you are on, but in the event of simplivity commands being inoperable, stop the SVTFS service or SVTFS@0 (3.7.10 and later), and then halt the OVC like any other Linux machine.

Or if this is a true emergency and you need to power everything down as fast as possible and all VMs are already off, just start halting the OVC using "shutdown -h now", since you don't have to worry about data loss to the VMs.

[Major88]

ok, i will try this solution.

what is the commande for stoping svtfs service?

---

stop svtfs don't working, for me, stop not recognized

i've Omnistack v4.0.0, the lastest.

[Major88]

ok,

it's workin for me
for second node this command

svt-shutdown-safe --emergency

it will ask for password of svtcli user

And for the last node

svt-shutdown-force --emergency

[DaveOb]

Just on the different methods to shutdown the OVC,

The safeshutdown from the GUI is the best method. 

It will wait for HA sync of the VMs if all VMs are in HA sync the OVC will shutdown,if not then the shutdown will pause waiting for HA sync and orderly stopping of services Thats why you have to use "force" option in some instances as this  system can never gain HA sync if multiple nodes are powered off in the same failure domain,

The svt-shutdown-safe svt-shutdown-force are essentially the same command as exposed in the GUI.

As mentioned you can use stop svtfs or systemctl start svtfs@0 ,This method does not wait for HA sync but will attempte to gracefully shutdown the OVC process  in an orderly manner.

You should look for the following lines in svtfs.log to show that the information in the TIA card was saved to disk 

2020-01-14T17:58:02.588Z INFO 0x7f732d065880 [:] [nvrammanager] nvram_manager.cpp:2137 Persisting complete NVRAM image to disk. Size 8237613056 8237613056
2020-01-14T17:58:10.432Z INFO 0x7f732d065880 [:] [nvrammanager] nvram_manager.cpp:2186 NVRAM image saved to hard disk.

Only then should the ovc be shutdown othewise you increase the  risk albeit small  of svtfs not restarting.

[jlangmead]

I'm running SimpliVity 4.0.1

I'm also finding the complete shutdown of a federation a very hit and miss affair. Everything is fine until I get to the last node that is running the VCSA. I shut all other vm's down with the exception of the VCSA and the OVC and then shutdown the VCSA. 

I next SSH into the OVC as SVTCLI (as the VCSA is shutdown) and this is where I hit issues. I get the same error on all six of the two node environments I've tried so far - so I must be missing a step.

I run

~$ svt-shutdown-force --emergency

This asks me to confirm I want to shutdown even without achieving HA compliance to which I respond 'yes'. The it asks me for the svtcli password (which I enter). it then returns 'A shutdown task is not running on the virtual controller' and nothing more happens. This did work on one node but not any of the others I tested with.

I have tried 'sudo su' followed by 'source /var/tmp/build/bin/appsetup' and 'stop svtfs' but this says "command 'stop' not found, but there are 18 similar ones"

Where do I go from here please?