Storage Boards Cleanup
To make it easier to find information about HPE Storage products and solutions, we are doing spring cleaning. This includes consolidation of some older boards, and a simpler structure that more accurately reflects how people use HPE Storage.
HPE StoreVirtual Storage / LeftHand
cancel
Showing results for 
Search instead for 
Did you mean: 

Rebalance Gateway Connections

jpiscaer
Advisor

Rebalance Gateway Connections

One of my customers is running a six-node Multi-Site Cluster with a single Virtual IP (VIP) for optimal load balancing.

 

I'm tasked with preparing a failover test. One of the scenario's I'm working on is simulating a power failure of the first site. This will result in a couple of things:

- Within a couple of seconds, all the Gateway Connections pointing to the now failed storage nodes will be assigned to the surviving storage nodes (i.e. the storage nodes on the secondary site)

- VMware HA will restart affected virtual machines on the surviving hosts (i.e. the hosts on the secondary site).

 

My main concern is this: after failover, all the Gateway Connections will be hosted by nodes on the secondary site. After I restore the primary site, I want to rebalance the Gateway Connections, so that the nodes on the primary site will be assigned half of the Gateway Connections. Other than the 'RebalanceVIP' CLI command, which seems to be highly unreliable, I do not know of any way to reassign Gateway Connections to storage nodes on the primary site Does anyone know of a reliable method to accomplish this?

18 REPLIES
5y53ng
Regular Advisor

Re: Rebalance Gateway Connections

Hi,

 

I asked a similar question a couple of months ago. I was told the gateway connections would rebalance themselves after ten minutes or so, but I have found this is not true after repeated tests of the same scenario you describe. The only reliable method I have found to change gateway connections is to reboot the remote VSAs (one at a time, wait for resyncs and check availaility tabs) to change up gateway connections. I know this isn't desireable, but it will rebalance your gateway connections for sure.

 

Maybe someone will chime in with a better method, which I would be interested in hearing.

Paul Hutchings
Super Advisor

Re: Rebalance Gateway Connections

I thought it just rebalanced itself?  We had a power outage that ran past our UPS time at one site and the nodes went down.  We brought them up and at some point later (I can't say I noticed when) all the connections rebalanced.

 

I can't see there would be another way beyond the CLI.

 

Paul

jpiscaer
Advisor

Re: Rebalance Gateway Connections

Would placing servers and nodes in separate sites (or not placing them in a separate site) make any difference? Servers and nodes are now places in two specific sites (with the FOM in a third logical site).

 

I now have a operational procedure in place to check the site affinity of a volume (by checking the Gateway Connection for each volume), grouping all VM's on a given volume into a VMware DRS VM group and linking the two together using a DRS vm-to-host rule.

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

You mention it is a multisite cluster... 3 nodes on each site...

 

I assume the nodes are placed in the 2 created sites...  Did you make the servers also site aware?

 

If so, they will only connect to the managers in their own site... The InterSite link will not be used for iSCSI trafic only replication trafic...

If the servers are not site aware, they will use all managers and so the ISL is also used for iSCSI trafic...

 

How many managers do you have running? 5 I assume?

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

I have 3 storage nodes per site, and placed them in two sites within the CMC.

I have a bunch of VMware ESXi hosts, also placed them in the two sites within the CMC.

 

How do the servers know they can or cannot connect to a storage node in the remote site? Is this handled by the iSCSI Portal (Virtual IP) and redirection?

 

I need to prevent a situation that all Gateway Connections are hosted on storage nodes on a single site: I do not have enough compute resources (i.e. ESXi-hosts) to run all VM's on the SAN on a single site, so I need to make sure the volumes are evenly spread across sites.

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

Hi,

 

The VIP IP address is active on all nodes where the Managers are running. Beneath there is a load balancing mechanism to distribute the incoming onnections...

 

If you make the servers also site aware, they will use only the managers in their own site... So you can control some kind of way already your trafic... Only if there are no managers running they will go searching in the other sites.

 

That is why it is important to have 5 managers running spread accross the 2 sites...

 

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

But how do I control the binding of Gateway Connection to a site (or more specifically, a storage node)? I need control Gateway Connections after I have done my failover tests. Because the number of hosts on the primary site is higher than the number of hosts on the secondary site, I need to make a 70/30 percent distribution of volumes between sites.

 

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

Hi,

 

Take a look where the managers are running... The nodes with the manager running will act as Gateway Connection also...

 

If you want more connections in 1 site, run more managers over there... So in your case it will be 3 in the biggest and 2 in the smallest site...

 

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

So by dismantling the logical third site (containing the FOM and the back-up server) and adding those to the primary site, the number of managers in the primary site is three (two normal managers and a FOM); this would make the number of managers in the primary site higher, and thus it would load balance the Gateway Connections unevenly?

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

You have a good point there. The FOM is there to decide which site must stay active in case of a split-brain situation on the network.

If you want to move the FOM to the first site, you can remove the FOM and start the manager on the third storage node... Result would be the same then...

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

That's exactly what I'm thinking: if the FOM is in the same site, there's no need for it anymore, just start a third manager in the primary site.

jpiscaer
Advisor

Re: Rebalance Gateway Connections

Can you inform me wether your systems (ESXi-servers) are grouped into sites, or not? I suspect your servers aren't grouped into a site, thus SAN/iQ has no idea that it needs to rebalance the Gateway Connections.

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

It depends from customer to customer... Some want site-aware servers, some others not... Performance versus high availability... A lot depends of the ISL between the sites...

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

I'm still struggling with what the Gateway Connections actually do in a Multi-Site SAN with site-affinity for ESXi-hosts.

 

An ESXi-host connects to the cluster's Virtual IP and gets redirected to one of the storage nodes. You can see which storage node a host connects to for each volume by viewing the Gateway Connection. Each volume is accessed through the same storage node, so all hosts connecting to a volume show the same Gateway Connection.

 

What other implications does the Gateway Connection have when connecting ESXi-hosts in a Multi-Site SAN (single Virtual IP, two sites)?

 

 

Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

If the servers are site aware they should connect to a node (with a manager running) in their specific site.

If the servers are not site aware they should connect to all nodes available...

 

My test/demo setup is currently single site so I cannot validate this though... If I have the occasion I will take a deeper look at it...

 

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

If hosts are site-aware, they connect to nodes in the same site; but does each host connect to a single storage node for a given volume?
In other words, does a host connect to multiple hosts for a single volume?
Bart_Heungens
Honored Contributor

Re: Rebalance Gateway Connections

If you install the MPIO DSM (currently available only for Windows) then yes the server will connect to all available storage nodes... If the MPIO DSM is not installed it will connect through the VIP...

 

Since the MPIO DSM does not exist yet for ESX, the only way to get the best performance is to create multiple VMkernel ports and bind them to the vmhba following the procedure that is already often mentioned here on the forums. Command line stuff in ESX4, via the GUI in ESX5...

 

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
jpiscaer
Advisor

Re: Rebalance Gateway Connections

yeah, not having a DSM for ESXi is a real downer at the moment.

I did a test this week: the storage node that is the Gateway Connection for a volume handles *all* I/O to/from the ESXi-host. The storage node will communicate with other storage nodes on the back-end to fetch / write blocks. This limits the maximum performance of a single volume to the maximum performance of a storage node. Because SAN/iQ assigns Gateway Connections to as much storage nodes as possible, this will load balance a bit, but it's a non-optimal configuration.

HP needs to release a DSM (or MPP) for ESXi to get performance for a vSphere environment on par with a Windows environment.