Rebalance Gateway Connections

[jpiscaer]

One of my customers is running a six-node Multi-Site Cluster with a single Virtual IP (VIP) for optimal load balancing.

I'm tasked with preparing a failover test. One of the scenario's I'm working on is simulating a power failure of the first site. This will result in a couple of things:

- Within a couple of seconds, all the Gateway Connections pointing to the now failed storage nodes will be assigned to the surviving storage nodes (i.e. the storage nodes on the secondary site)

- VMware HA will restart affected virtual machines on the surviving hosts (i.e. the hosts on the secondary site).

My main concern is this: after failover, all the Gateway Connections will be hosted by nodes on the secondary site. After I restore the primary site, I want to rebalance the Gateway Connections, so that the nodes on the primary site will be assigned half of the Gateway Connections. Other than the 'RebalanceVIP' CLI command, which seems to be highly unreliable, I do not know of any way to reassign Gateway Connections to storage nodes on the primary site Does anyone know of a reliable method to accomplish this?

[5y53ng]

Hi,

I asked a similar question a couple of months ago. I was told the gateway connections would rebalance themselves after ten minutes or so, but I have found this is not true after repeated tests of the same scenario you describe. The only reliable method I have found to change gateway connections is to reboot the remote VSAs (one at a time, wait for resyncs and check availaility tabs) to change up gateway connections. I know this isn't desireable, but it will rebalance your gateway connections for sure.

Maybe someone will chime in with a better method, which I would be interested in hearing.

[Paul Hutchings]

I thought it just rebalanced itself?  We had a power outage that ran past our UPS time at one site and the nodes went down.  We brought them up and at some point later (I can't say I noticed when) all the connections rebalanced.

I can't see there would be another way beyond the CLI.

Paul

[jpiscaer]

Would placing servers and nodes in separate sites (or not placing them in a separate site) make any difference? Servers and nodes are now places in two specific sites (with the FOM in a third logical site).

I now have a operational procedure in place to check the site affinity of a volume (by checking the Gateway Connection for each volume), grouping all VM's on a given volume into a VMware DRS VM group and linking the two together using a DRS vm-to-host rule.

[Bart_Heungens]

You mention it is a multisite cluster... 3 nodes on each site...

I assume the nodes are placed in the 2 created sites...  Did you make the servers also site aware?

If so, they will only connect to the managers in their own site... The InterSite link will not be used for iSCSI trafic only replication trafic...

If the servers are not site aware, they will use all managers and so the ISL is also used for iSCSI trafic...

How many managers do you have running? 5 I assume?

Kr,

Bart

[jpiscaer]

I have 3 storage nodes per site, and placed them in two sites within the CMC.

I have a bunch of VMware ESXi hosts, also placed them in the two sites within the CMC.

How do the servers know they can or cannot connect to a storage node in the remote site? Is this handled by the iSCSI Portal (Virtual IP) and redirection?

I need to prevent a situation that all Gateway Connections are hosted on storage nodes on a single site: I do not have enough compute resources (i.e. ESXi-hosts) to run all VM's on the SAN on a single site, so I need to make sure the volumes are evenly spread across sites.

[Bart_Heungens]

Hi,

The VIP IP address is active on all nodes where the Managers are running. Beneath there is a load balancing mechanism to distribute the incoming onnections...

If you make the servers also site aware, they will use only the managers in their own site... So you can control some kind of way already your trafic... Only if there are no managers running they will go searching in the other sites.

That is why it is important to have 5 managers running spread accross the 2 sites...

Kr,

Bart

[jpiscaer]

But how do I control the binding of Gateway Connection to a site (or more specifically, a storage node)? I need control Gateway Connections after I have done my failover tests. Because the number of hosts on the primary site is higher than the number of hosts on the secondary site, I need to make a 70/30 percent distribution of volumes between sites.

[Bart_Heungens]

Hi,

Take a look where the managers are running... The nodes with the manager running will act as Gateway Connection also...

If you want more connections in 1 site, run more managers over there... So in your case it will be 3 in the biggest and 2 in the smallest site...

Kr,

Bart

[jpiscaer]

So by dismantling the logical third site (containing the FOM and the back-up server) and adding those to the primary site, the number of managers in the primary site is three (two normal managers and a FOM); this would make the number of managers in the primary site higher, and thus it would load balance the Gateway Connections unevenly?