Shutting down a complete federation

[jlangmead]

I've started a new thread on this rather than hijacking an old one :0)

I'm running SimpliVity 4.0.1

I'm finding the complete shutdown of a federation a very hit and miss affair. Everything is fine until I get to the last node that is running the VCSA. I shut all other vm's down with the exception of the VCSA and the OVC and then shutdown the VCSA. 

I next SSH into the OVC as SVTCLI (as the VCSA is shutdown) and this is where I hit issues. I get the same error on all six of the two node environments I've tried so far - so I must be missing a step.

I run

~$ svt-shutdown-force --emergency

This asks me to confirm I want to shutdown even without achieving HA compliance to which I respond 'yes'. The it asks me for the svtcli password (which I enter). it then returns 'A shutdown task is not running on the virtual controller' and nothing more happens. This did work for one cluster  but not any of the others I tested with.

I have also tried 'sudo su' followed by 'source /var/tmp/build/bin/appsetup' and 'stop svtfs' but this says "command 'stop' not found, but there are 18 similar ones"

Where do I go from here please?

[DaveOb]

svt-shutdown-force  must be proceeded by the svt-shutdown-safe  command, the  svt-shutdown-safe command initiates the shutdown task while svt-shutdown-force makes the command bypass the HA task.

The newer versions of the OVC implement a later kernel version and the stop svtfs command is deprecated in favour of systemctl status svtfs@0

When stopping svtfs via the cli I would advise checking for the following line in the $SVTLOG especially if the server is going to be powered off for an extened period.

grep -A1 -B1 " NVRAM image saved to hard disk."

2020-03-16T04:45:59.984Z INFO 0x7fd896d66f00 [:] [nvrammanager] nvram_manager.cpp:2186 NVRAM image saved to hard disk.

[jlangmead]

Thanks Dave

That's clear now - thanks. I wasn't aware that svt-shutdown-force had to be preceeded by the svt-shutdown-safe command. I think the cluster that did work fine was one where I did do the safe command first but that was just coincident - The others I only did the force command as it was the last of the OVC's.

These are likely to be down for extended periods - perhaps several weeks as they are currently under test and not in production. Can you just clarify if I should only need to check the $SVTLOG if I stop the SVTFS? If I just issue the svt-shutdown-safe followed by the svt-shutdown-force commands I can be confident the NVRAM has been written ok?

thanks

[DaveOb · Accepted solution]

svt-shutdown-safe will stop all services gracefully.

svt-shutdown-force will attempt skip the HA portion of the shutdown (which cannot be achived when a single node in the cluster is left running) but also if the services themselves are hanging.

hanging shutdowns are uncommon especially in the latest versions. 

But to be extra sure I would check the for the nvram being flushed to disk if the svtfs process is stopped via cli or via the svt-shutdown-force command.

If it is not in 99% of cases it will be fine as the data  is stored in the Accelerator card, however if the system is left for long periods  (multiple days)the card looses the hold up power and the info may be lost similar to the cache on a raid controller.

[Stefano Colombo]

Hello , I'm facing the same issue

I'm trying to shutdown the federation completely in a condition where the vcenter is already shutdown

We have a 3 node cluster so I tried the following

1) ssh on the first OVC using svtcli user.

using sudo and source command

performed shutdown . It worked

2) ssh on second node with svtcli user

done the same procedure as the first node but it hang checing for HA .

tried shutdown -force did not help

svt-shutdown does not work as it expect the vcenter

dsv-shutdown-start --force did not help either

Can anyone tell me the correct procedure to shut the federation down ?

thanks

[Stefano Colombo]

Hello

I didn't notice the option --emergency that solved my question

thanks