HPE SimpliVity
cancel
Showing results for 
Search instead for 
Did you mean: 

Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

 
SOLVED
Go to solution
Highlighted
Occasional Advisor

Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

Hello,

I am looking for best pratics (procedure) for correct shutdown of 3 nodes Simplivity cluster v.40.0.

Firstly i shutdown all VMs exept vCenter

I shutdown one of the virtual controller by the Simplivity menu in the vcenter plugin. - it's ok.
Maintenance mode -is ok too.

But when i shutdown 2nd virtual controller, the vcenter is became unresponsive. And at this moment i can't login to ssh session with vcenter autentification to use svt-shutdown-safe.

and i blocked it this state. If is there other solution for completly shutdown the cluster's node with hosted vcenter (vcsa)

6 REPLIES 6
Highlighted
HPE Pro

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

This is happening because the nodes on which VC  is stored have both been powered off,

https://damianerangey.com/ for a good explanation of how vm copies are managed.

You should be able to identify  from the DRS rule the location of the VC copies and power off the node without any VC copy first then a node with one copy.

This should leave you with one OVC running and VC.

Put 2x nodes into maintence mode and shutdown.

Power off VC gracefully.

SSH to the last remaining  OVC and issue a svt-shutdown-force command,

ssh to esxi or connect directly and shutdown,

Alternately you can just shutdown all VMs including VC and ssh to all 3 OVC and use the svt-shutdown-safe command and shutdown the ESXi via connecting directly.

 


I am an HPE employee
Accept or Kudo
Highlighted
Occasional Advisor

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

i can connect to ssh, but without vcenter i can only login as svtcli user, and svtcli user can't use the svt commands:

i've this message


ERROR [47]: Missing session credentials.
Highlighted
Advisor
Solution

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

Try a --emergency on the end of the command. Some commands have that built in to ignore the session credentials requirement. I don't know what version you are on, but in the event of simplivity commands being inoperable, stop the SVTFS service or SVTFS@0 (3.7.10 and later), and then halt the OVC like any other Linux machine.

Or if this is a true emergency and you need to power everything down as fast as possible and all VMs are already off, just start halting the OVC using "shutdown -h now", since you don't have to worry about data loss to the VMs.

Highlighted
Occasional Advisor

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

ok, i will try this solution.

what is the commande for stoping svtfs service?


stop svtfs don't working, for me, stop not recognized

i've Omnistack v4.0.0, the lastest.

Highlighted
Occasional Advisor

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

ok,

it's workin for me
for second node this command

svt-shutdown-safe --emergency

it will ask for password of svtcli user

And for the last node

svt-shutdown-force --emergency

 

Highlighted
HPE Pro

Re: Gracefull shutdown 3 node cluster with vcenter (vcsa) hosted in the one of this node

Just on the different methods to shutdown the OVC,

The safeshutdown from the GUI is the best method. 

It will wait for HA sync of the VMs if all VMs are in HA sync the OVC will shutdown,if not then the shutdown will pause waiting for HA sync and orderly stopping of services Thats why you have to use "force" option in some instances as this  system can never gain HA sync if multiple nodes are powered off in the same failure domain,

The svt-shutdown-safe svt-shutdown-force are essentially the same command as exposed in the GUI.

As mentioned you can use stop svtfs or systemctl start svtfs@0 ,This method does not wait for HA sync but will attempte to gracefully shutdown the OVC process  in an orderly manner.

You should look for the following lines in svtfs.log to show that the information in the TIA card was saved to disk 

2020-01-14T17:58:02.588Z INFO 0x7f732d065880 [:] [nvrammanager] nvram_manager.cpp:2137 Persisting complete NVRAM image to disk. Size 8237613056 8237613056
2020-01-14T17:58:10.432Z INFO 0x7f732d065880 [:] [nvrammanager] nvram_manager.cpp:2186 NVRAM image saved to hard disk.

Only then should the ovc be shutdown othewise you increase the  risk albeit small  of svtfs not restarting.

 

 

 

 

 

 


I am an HPE employee
Accept or Kudo