Query: Failure scenario - VMs moved/started on main node after link failure - network

[SPTTSM]

We have recently done a DRP test whereby we disable the 10G link betwen the nodes and then note what happens.

Node A = Main Node
Node B = offsite Node

After closing the 10G link we noted that the VMs that were on node B were eventually activated/registered with the main node, this is exactly as we expected things would happen.

However neither of our 2 test VMS managed to successfully become available the local network.

The VMs were definately up on Node A, one used DHCP and obtained no address, the second had a fixed IP but was not pingable/reachable.   They are both in standard VLANs, and if migrated manually function without any problem.

If there a stage which we did not apply correctly, do the VMs have to be rebooted,  or any other reason for the networks not being accessible.  Is this normal practice ?   

Also, when the Node B comes back online is it normal that the VMs are nor returned to Node B ?  I can understand that this might be normal practice , we just havent found the documentation which clearly states how the scenario should unfold and refold when back online..

Thanks in advance

[gustenar]

Hello @SPTTSM 

Are you using any tool to perform the migration, something like Rapid-DR? 

What features do you have enabled in vCenter? vSphere-HA, DRS?

[SPTTSM]

Hi, 

I am not aware that we use any solution other than the standard Simplivity 2 Node + Arbiter setup.

[Sanika]

Hello @SPTTSM 

Considering what you've described, I believe your query relates to Rapid DR, specifically DRP test (Disaster Recovery Plan).
Looks like the DRP test was succesful partially. The VMs migrated to Node A as expected but they experienced network connectivity.

In my opinion, VLAN restrictions or specific DHCP settings may prevent Node A's DHCP server from assigning addresses to migrated VMs. Firewall rules on Node A may also block traffic to or from the migrated VMs. Also make sure the IP addresses on the VMs are within the correct network range & VLAN.

And about VMs not returning to Node B, this could be due to insufficient resources on Node B or VM affinity rules i.e if VMs are set to remain on Node A, they might not automatically migrate back when Node B comes online.

Apart from this, here are some additional resources regarding RapidDR that you might find helpful:

HPE SimpliVity RapidDR 3.6.0 User Guide - https://support.hpe.com/hpesc/public/docDisplay?docId=a00117362en_us&docLocale=en_US

HPE SimpliVity - Testing the failover on Rapid DR Recovery Plan - https://support.hpe.com/hpesc/public/videoDisplay?videoId=vtc00030716en_us

HPE SimpliVity - Testing the failback on Rapid DR Recovery Plan - https://support.hpe.com/hpesc/public/videoDisplay?videoId=vtc00030732en_us

HPE SimpliVity - Executing failover Recovery Plan on RapidDR - https://support.hpe.com/hpesc/public/videoDisplay?videoId=vtc00030748en_us

HPE SimpliVity - Executing failback Recovery Plan on Rapid DR - https://support.hpe.com/hpesc/public/videoDisplay?videoId=vtc00030749en_us

Hope this information helps.

Regards,
Sanika.

If you feel this was helpful, please click the KUDOS thumb below. Also consider marking this as an "Accepted Solution" , if the post has helped to solve your issue.

[SPTTSM]

Hi Sanika, 

Ok I can now confirm the following, we dont have Rapid DR and DRS is disabled on the Cluster.  All Vms are on the same VLANs, or at least on VLAns that can communicate between each other.  There are no Firewall rules in place.

We did a new test this morning, when the link goes down the VMs are transfered to NODE A without problem.  So no problesm here.

The VMs that are transfered cannot be PINGed from network within the same VLAN.
BUT 
The VMs that are transfered can PING any other VMS on the network with the same VLAN..  

It's as if the the Simplivity is blocking incoming traffic to these VMs but allowing outgoing traffic, or there are routing tables that are not being cleared ( ARP cache). 

When the link comes back up , the VMs remain on NODE A and all the networking functions correctly for these VMS again without us having to change anything.  

Here is a recap in Visio.

[gustenar]

@SPTTSM 

It looks like the two nodes are connected to each other using a 10Gb cable. Check the NIC teaming mode in your vSwitches, make sure it is set to Active/Passive. Here's the SimpliVity networking best practices guide, check page 11 that talks about direct connected clusters and make sure all your network settings are correct:

HPE SimpliVity for vSphere networking best practices 

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you have no further query, and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[SPTTSM]

@gustenar @Sanika Sorry for not replying sooner, I have been on vacation.

Unfortunately we have not resolved this issue, everything appears to be correctly configured relatiing to the network, we cannot find any problem there.

We have VMs running on both nodes on a daily basis and have no problem at all..

Question ; Can you please confirm the following : When one of the nodes fail should the VMs from the failed node automatically become active on the Active Node ?   Is this how things should work or are we requried to use other solutions to get things running ?

( I know that this takes around 5 minutes which is perfectly acceptable - the Arbiter needs to do it's work).

[gustenar]

Ok there are two possible scenarios. 

1. The OmniStack controller (OVC) has a problem and goes down. In this scenario the Storage IP of the OVC performs a failover and another node in the cluster will take care of the storage access. The VMS continue to run normally in the same host. 

2. The host crashes unexpectly. If you have Vmware's vSphere HA configure and running the VMs will restart in another node in the event of a node failure. Check VMware's documentation for more details on functionality and settings. This is not a SimpliVity function.