Simplivity "svt-shutdown-safe" fails in a three node cluster when trying to shutdown o

[TomiKarlsson]

Hi all. I have a situation where I have to move 3 cluster nodes from one DC to another. Live migrated VMs to the other nodes and first node was safely shutdown and moved to new DC and rejoined the cluster with no problems.
Now, the third and last node/OVC refuses to shutdown safely, because I just found out that the OVC cannot ping with Jumbo packets one of the already moved nodes, so IP failover isn´t succesful. The Jumbo packets don´t travel from one datacenter the new datacenter and our networking guys have checked all settings correctly in core switches. Only packet size 8920 goes through:
administrator@vsphere@omnicube-ip115-22:~$ ping -s 8920 10.255.144.75
PING 10.255.144.75 (10.255.144.75) 8920(8948) bytes of data.
8928 bytes from 10.255.144.75: icmp_seq=1 ttl=64 time=1.34 ms
8928 bytes from 10.255.144.75: icmp_seq=2 ttl=64 time=1.31 ms
8928 bytes from 10.255.144.75: icmp_seq=3 ttl=64 time=1.32 ms
8928 bytes from 10.255.144.75: icmp_seq=4 ttl=64 time=1.29 ms
8928 bytes from 10.255.144.75: icmp_seq=5 ttl=64 time=1.28 ms
8928 bytes from 10.255.144.75: icmp_seq=6 ttl=64 time=1.28 ms

Is there any option to force OVC to check the Storage and Federation network with lower MTU??? Shutting down VMs is not an option for me. Output from the netcheck.sh tool/script which I once (thanks to support!) received to check the Federation networking:

Testing from OmniStackVC-192-168-115-22 has completed and errors have been found; the following tests have failed:
---------------------------------------------------------------------------------------------------------------------------------------------------------
Federation Interface:10.255.144.75:Failed to ping with MTU 8972 please check vswitch1 MTU Size
Storage Interface:10.255.133.71:Failed to ping with MTU 8972 please check vswitch1 MTU Size
Federation Interface:10.255.144.77:Failed to ping with MTU 8972 please check vswitch1 MTU Size
Storage Interface:10.255.133.75:Failed to ping with MTU 8972 please check vswitch1 MTU Size

I don´t dare to use "svt-shutdown-force". Anyone, any ideas??? All help much appreciated!

Best regards, Tomi Karlsson

[dhooley]

Hi Tom,

First off I would be recommending opening a support ticket for this issue as there are many factors/variables in this type of scenario. Very hard to give you a detailed answer via this forum.

If there are currently no VM's running on the third node (should not be seeing as it will be shutdown) then running the svt-shutdown-force command is safe. You can check this via vCenter also of course.

My recommendation would be to check for any HA issues on the VM's using "svt-vm-show --violations" on the OVC VM. If all running VM's are green then the svt-shutdown-force command is the best action.

If you are anyway unsure on the above then please open a support ticket.

Hope this helps!

[TomiKarlsson]

Hi and thanks for your answer!

Yes I opened a ticket for this 2 days ago. Right now I was also thinking the same that since we now have Live migrated all running VM´s to the new datacenter and all VM´s show HA state OK, it should be safe to use "svt-shutdown-force", without any impact on running VM´s on the other 2 nodes, right?

I´ll also attach a picture of the datacenters and cluster state currently.

Best regards, Tomi

[dhooley]

Hi Tom,

Yes, assuming that the HA sync check returns all green then there will be no impact of shutting down the remaining OVC.

The force flag as I mentioned simply removes the HA check when stopping the SVTFS (storage service) of the OVC pre-shut down.

The only thing that could be currently on the remaining node is Secondary replicas. For any VMs with Secondary replicas on this node will go out of HA sync until the move is complete. This is a common occurrence when doing maintenance such as this.

The case owner will be able to confirm this also, if you want to share the case number I can ensure that you are in safe hands

Thanks!

[TomiKarlsson]

Thanks! Case number is: HPE Support Case 5339668794

Yes, I´m aware of the "svt-shutdown-force" command. Interesting thing was that I don´t get any output from "dsv-balance-show"? Only from "dsv-balance-manual -q", which shows:

index Node IP                      Pri      Sec Total
node 1         192.168.115.21     15        6      21
node 2        192.168.115.22      7        10     17
node 3        192.168.115.23     11        17     28

Node 2 is the one with no running VM´s. It shows that it has 7 Primary Replicas but shouldn´t be a problem, right?

Br, Tomi

[dhooley · Accepted solution]

There should be 0 Primary replicas on the node that you are using svt-shutdown-force on. Otherwise there is a good chance of DU.

I just chatted to the case owner and looks like he squared this away with you. Good to hear!

[TomiKarlsson]

Hi again! Yes, there was 7 Primary Replicas still on the node to be shutdown. Support instructed to vmotion the VM´s to the third already located in the new datacenter and then the "dsv-balance-manual -q" showed 0 Primary Replicas remaining on the node to be shutdown!
"svt-shutdown-force" was issued and OVC was nicely shutdown without any impact to running VM´s

The node is now on it´s way to the new datacenter, soon to be back in the cluster

Thanks a lot for your assistance! A big thanks also to Keith Gorman from support who handled the case!

May the "--force" be with You!!!

Best regards, Tomi

[KGman]

Glad to hear it Tomi and thank you for the kudos!