StoreVirtual Storage
1748201 Members
3088 Online
108759 Solutions
New Discussion юеВ

Re: P4500 Multi site, FOM and network isolation

 
akachrmol
Occasional Visitor

P4500 Multi site, FOM and network isolation

P4500 Multi site, FOM and network isolation

 

Hi,

IтАЩm having a P4500 cluster with 10 nodes тАУ 5 on each site, in a multi-site cluster.

We are now talking about implementing a Failover manager in a third site so the new setup will look like this.

SiteA, 5 nodes, 2 managers

SiteB, 5 nodes, 2 managers

SiteC, 1 FOM

All three sites are connected тАУ but one scenario that I canтАЩt figure out is: What happens if the network between SiteA and SiteB goes down. SiteC can still see SiteA and SiteC тАУ but SiteA and SiteB canтАЩt see each other.

Will this keep one site up? Or will both stop I/O ?

8 REPLIES 8
oikjn
Honored Contributor

Re: P4500 Multi site, FOM and network isolation

1.  thats a good question and I"m not sure of the answer and haven't tested it personally.

2.  Really, JUST thinking about a FOM site?  Really?  Thats scarry if this is a real multi-site installation already in production.

3.  I think the assumption is that there is a backup link so that you never run into this situation.  One way you can avoid this situation is that if you have A<->B, A<->C, B<->C.  Where C is the FOM site, you can usaully route A<-C->B with a lower priority than the A<->B link so that you can never run into a situation where there is communication between both sites and the FOM and yet no communication between the two sites. At that point you might have to manually shut down one site if your FOM site bandwidth isn't sufficient to handle your load, but it will keep everything running like normal.

 

HP makes multi-site clusters sound easy to impliment and in truth setting it up w/ CMC IS easy... almost too easy, but having the support infrastructure setup to match what is required for a multi-site cluster is typically beyond the capabilities of most corporations. 

akachrmol
Occasional Visitor

Re: P4500 Multi site, FOM and network isolation

2. Yes it is in production but we have chosen to use тАЭVirtual ManagerтАЭ for deciding which site to bring up in the event of a failure.

3. I agree.

Since we are in production it is not easy to test тАУ but it would be nice to know what to expect. J

 

 

oikjn
Honored Contributor

Re: P4500 Multi site, FOM and network isolation

If I had the time, I would spool up a bunch of trial VSAs in a VM environment w/ virtual networking to test this, but I personally don't have the time and won't for a long time.  The nice thing about the VSA software and trials is that they let you play with every feature you want so if you really cared about this you could setup 4 small VSAs and a FOM and test this yourself without risking the production SAN.

 

a virtual manager can be a nice temporary stop-gap, but if you truely need a multi-site cluster, then you probably can't afford the downtime generated from the loss of that virtual manager during a site failure.

RemyZ
Advisor

Re: P4500 Multi site, FOM and network isolation

Hmm. The job of the FOM is to notice that nodes (or sites) can or cannot see each other anymore. When it notices that nodes in site A cannot see nodes in site B, it will make volumes on one of the sites unavailable.

 

One of the requirements however, is that site C has a direct connection to site A and another direct connection to site B. Otherwise you'll have an single-point-of-failure.

 

--------------------------------------
Remy Zandwijk
VU University Amsterdam
HPstorageTom
HPE Pro

Re: P4500 Multi site, FOM and network isolation

Well, the job of the FOM is to solve exactly such split-brain situations. So what will happen is, that one of the sites (either A or B) will be kept up while the access to the other site will be blocked.

oikjn
Honored Contributor

Re: P4500 Multi site, FOM and network isolation

tom,

so does it just use a random number generator to decide which site to take offline?  I get the FOM should prevent any split-brain function, but it sure would suck if your primary site goes offline becuase it loses communication with your DR site.

RemyZ
Advisor

Re: P4500 Multi site, FOM and network isolation

If you did configure a primary site, then the FOM would always choose the primary site to stay alive (in the case both sites cannot see eachother anymore).

--------------------------------------
Remy Zandwijk
VU University Amsterdam
akachrmol
Occasional Visitor

Re: P4500 Multi site, FOM and network isolation

I will try to make a test to see what happens if the site interlink goes down with no majority in the two sites. It would be nice to see how FOM will handle this.