Best way to convert SAN from multisite to single site

[ymaxwell]

We have the older hp storageworks p4500 g2 models deployed in a multsite SAN mode across 2 phyiscal datacenters and 2 management groups (1 cluster per group). We've lowered out utilization to the below details

1. management group 1
   1. 5 nodes in each datacenter (site A, site B)
   2. all but 2 LUNs are configured as 2 way network RAID level 10
   3. 91 TB total space, 56TB available space, 23 TB used space (rounded up). 38% utilization
2. management group 2
   1. 3 nodes in each datacenter (site A, site B)
   2. all LUNs are configured as 2 way network RAID level 10
   3. 55 TB total space,  28 TB available space, 27 TB used space. 49% utilization 

We want to pull nodes from site B and either create another cluster that can be presented to our hypervisors located in site B OR just have em to the side as spare parts when drives fail since we no longer have support until our management gets through their lonnnnnng review process . in short we'd be effectively treating our nodes as local SANs so that we're no longer writing data over our 1 gbps circuits.

What is the best approach you guy's would recommend? 

We tried to start removing storage systems (nodes) in Site B and  received an error stating "The operation cannot be performed because the cluster wou;ld not have the same number of storage systems at each site. We don't want to remove nodes in both sites just to get around that and have to re-add some of the same nodes in Site A due to all the re-striping.

The only option we see is to delete the entire Site B and hope that doesn't cause too much re-striping. We also are hoping that since we are under 50% utilization in both clusters we'd only see an increase of 50% once we remove all the nodes in Site B, which is half of the storage available right now.

[oikjn]

a multi-site cluster really requires that you have exactly equal performance/capacity/nodes on both sides.  Anything else and you would have to go async replication which you can then use different node combinations but the writes would always have to be from the one active site.

Do you require a performance isolation between the clusters or is there another reason you would require either two clusters or two management groups for DR?  If not, then moving to a single multi-site cluster should work, but its going to require you migrate the LUNs across the clusters first and then once everything is off that smaller management group you could delete it and add however many nodes you wanted to the other one.  It would be a pain in the ass, but doable and with only a bit of downtime as you could use remote replication to sync the LUNs then do a planned failover to the "remote" site. 

You can have a mixed management group where one cluster is multi-site and another is only on a single site.  The site information/logic is really 100% logic only and you can mess with them using the settings for the sites for each node.  we use a "multi-site" cluster setting within our server room to ensure that the data is not on the same rack instead of the same datacenter like a real multi-site cluster is.  We could have done that or we could have just played with the positions of the nodes within the cluster, but we decided on the multi-site method to make sure we didn't screw it up accidentally later.

the bottom line is that as long as a cluster is multi-site, you cannot have a disproportionate number of nodes on either side as both sides are supposed to be 100% equal.  If that isn't what you want, you really don't want a multi-site cluster.  I would talk to support to confirm, but my guess is if you change the site assignments to the nodes, you MIGHT be able to avoid a restripe as long as you plan the order of the nodes correctly so that the logical strips on the nodes don't change as a result of the change in site assignments.  They should be able to predict that one for you.

[ymaxwell]

Thanks for the reply oikjn and thorough explanation of how the multisite features work. The performance requirement we have is based on our point 2 point circuit utilization and part of modifying our data centers to meet our growing needs are only performing iSCSI read and writes over layer 2 in local datacenter traffic. We will also be using our lefthand nodes primarily for legacy file servers and and backup storage moving forward until we refresh or move to other solutions.

Getting away from the multisite option is exactly what we want, as you described, and now i realize that the errors we saw from reomving the nodes as we attempted is pretty much a safeguard of the system to adhere to the multisite design. We never really wanted a multsite option; this entire setup was done by a manager service company when our company was a tenth of its current size and our  I overstated the need to avoid re-striping data and its pretty much unavoidable at this point no matter how we try to solve this.

Regarding moving all LUNs off 1 of our clusters temporarily to modify it, that would work for our smaller cluster but not for our larger cluster as it has more utilized space than we even have available in the smaller cluster. We also won't be able to get any downtime for another quarter for the handful of baremetal servers pointing at 3 large LUNs as they run mission critical systems our business won't allow us to modify (unless they crash and burn on their own).  As a workaround would we not be able to simply delete a single site within the cluster? I would think that only removes half of the storage systems (nodes) from the cluster and of course removes site x, while converting the entire cluster from multi site to single site. I would think that would also leave our data intact so long as the remaining storage systems have enough total capacity to house all of our LUNs, with perhaps with some serious re-striping. Am I dreaming by suggesting that deleting 1 site in multisite mode (converting to single site) is the simplest solution here?

When I goto edit site in the CMC, select half of the nodes to remove, and click OK the prompt I am greeted with is 

"The operation will cause cluster "insert name" to change from a multi-site cluster to a standard cluster. Are you sure you want to change the cluster type?"

[oikjn]

You don't really mention if the servers connecting to the san are at one site or both sites.  If you are accessing all these LUNs active from both sites, you are much better off using the SAN in multi-site mode unless the usage on site B is going to be really really minimal. That way the connected servers can be site aware and only read/write from the local nodes instead of from the remote nodes.  There is a significant advantage to that site aware nature which if you have active servers on both sites you really should keep the cluster multi-site.

Converting a cluster between single and multi-site is really simple and easy to do since it only requires you to change the assignment of the nodes' sites... you would simply change all the cluter's nodes to be the same site and it will make it a single site cluster.  If you have a support contract, I would call HPE to frind out if this will require a node restripe...  if you don't care about time, you can always just do it and it will restripe and work out out, but keep in mind that the moment you do that and it does want to restripe, that is going to mean that the stripes will no longer be alligned to the physical sites and if you lose connection to a site you will lose read/write access on that entire cluster.

Best to migrate as many of the LUNs you can away from the cluster before doing anything since it will avoid their risk of downtime and also shorten the restripe process.

If the cluster really won't be used multi-site and the two locations are within reasonable driving distance I would suggest the following... (1) move every LUN you can to MG1.  (2) convert every LUN from NR10 to NR10+2. (3) shutdown every node at SiteB and physically move them to SiteA. (if you have a VIP for siteB, remove it first and migrate  (4)  start them up at physical siteA and change their IPs. (5) edit LUNs to be back to NR10 (6) change sites on nodes to be all at a single site (7) remove unneeded nodes.

That will take a while do complete, but will ensure you always have an active mirror copy of your mission critical data as you do not sacrofice availability even temporarilly.

[ymaxwell]

My apologies, this storage is being connected to by hypervisors and a few bare metal MS servers remotely over a 1 gig point to point circuit. The latency is not great and before we reduced our LeftHand utilization to current state the circuit utilization was pretty heavy due to the multisite config. We will be getting upgraded to 10 gig circuits eventually but that has not been approved by our business and we know providers typically give a 90 day window for provisioning in our area. 

We don't have active support contracts anymore which is why I came to this forum to bounce some questions off you and others who would have more insigh that myself and colleagues. You are right that our options will be time killers in either case because the re-stripes will take quite some time and so will the storage vMotions in ESXi 5.5. We will take most of the advice you suggested with a caveat from a new development this morning.

Our action plan-

1. We have a 3rd mgmt group/cluster (MG3) in SIte A that is in single site mode and has 19TB capacity. We were able to work with a product team to datawarehouse ALL of the data stored on VMs so that will be done shortly and we can create new LUNs with RAID 6 double parity to have some protection but maximize the space, and storage vMotion the data from MG2 to MG3 cluster to these new RAID 6 LUNs, move MG2 cluster from multi site mode to single site by changing node assignments, and pulling out the MG2 nodes from Site B so that we can have some spare parts. 
2. For our remaining MG1 cluster we'd probably just change node assignment so that we can convert it to single site mode and only keep that data on nodes physically located in Site A.
3.  Lastly we'd create a new cluster for Site B out of the remaining nodes physically located there and that will only be for backing up large databases (relational and flat file) that will be full state and deltas so thjey will no longer be written over our small circuits in short overnight windows.

Again we know that all of this will take some time but seems to be very worth it considering our lack of support and what we're using these SAN nodes for now (backups, legacy file servers).

[oikjn]

If I understood you correctly, there are active servers on both sites which need active connections to the LUNs.

If that is the case, a multi-site cluster is likely the BEST setup for you.  Assuming you have configured the server initiators in CMC to allign with their correct sites, this significantly reduces the WAN link bandwidth since all reads would be done locally and the only cross-site traffic would be the write replication and confirmations.

The only way a single site setup would make sense would be if your data is like 90+% write and then the read benefit is minimized.

The only problem I see with your plan is that step 2 isn't possible unless you mean that you will split the cluster apart so that instead of one cluster with nodes from siteA and siteB (physical, not logical), that you then make one cluster with nodes only from siteA and another cluster with nodes only from siteB.  That would be the only way you could make sure that the data only went to the local nodes.

I would suggest that unless you have a serious requirement for many management groups, you combine them while you have one empty.  Its much easier moving LUNs across clusters LIVE within a management group instead of the headache that is required to move a LUN from one management group to another.

The idea in #3 is really the only way you can actually address the bandwidth issue beyond simply "buying more bandwidth" (assuming you have actually setup your initiators on the SAN correctly and they are also assigned to their correct sites).  You should definitely focus on deciding what data requires an active connection across BOTH sites and keep that data on a multi-site cluster.  Anything remaining should be identified to determine which site the data should be active on and then create a single site cluster at both locations where you can then use each for local access and then remote replication for async-snapshot replication between the two for DR.

Ignoring the MG3 for now, I would suggest the end result of your original MG1+MG2 situation would best end up with a single MG1 with all multi-site LUNs in a cluster with 10 nodes in it, a cluster at siteA with 2 nodes, and a cluster at siteB with 2 nodes.  This leaves you with a spare node at each site and a "local only" cluster location for each site where you can move everything possible to reduce your site-site bandwidth requirements.