StoreVirtual Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

change site configuration - move LH nodes to different site

pirx4711
Frequent Advisor

change site configuration - move LH nodes to different site

Hi,

 

we have one management group with 2 multi site clusters (12 + 8 nodes). For reasons I don't remember anymore the 8 LH nodes in the second cluster - which was created appr. one year after the first one - were added to 2 new sites. Theses sites are at the same location/datacenter as the first 2, so this is a bit pointless I think. No server were added to these new sites. So I have one site with 12 nodes and 10 VMware hosts and a different site with 8 nodes and no server.

 

I think this is not right and want to remove the 8 nodes from their current sites and add them to the sites that were initially created.

 

But when I try to remove one (or all) of the node from their current sites a warning appears that this will cause the cluster to change from a multi site cluster to a standard cluster.

 

Can I safely ingnore this warning, because I'll add these nodes to the 2 remaining sites right after that? Will the cluster begin to restripe?

 

I'm not quite sure what to do, I would like to get rid of these 2 unnecessary sites.

12 REPLIES
oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

yea.  multi-site / single site is more a logical difference and you can switch clusters between the two back and forth if you wanted.  The only thing is with multi-site it requires a FOM at a 3rd site and that requirement is not there for a single site.

 

when playing with VSAs to undertand how everything worked, I frequently moved node sites around to see what happened under various scenarios (one of which was changing/removing sites).  removing one site converts it to a single site cluster which DOES generate a cluster re-stripe as the LUN redundancy structure adjusts its internal logic.

pirx4711
Frequent Advisor

Re: change site configuration - move LH nodes to different site

Just be sure I understodd you correct. Removing nodes from one multi-site and adding them to another one does not trigger a restipe. But removing a site would?

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

any time you change the site on a node a restripe of the cluster group with that node will occur.

 

That said, there will be no data loss during the restripe.  You can change sites, add sites, remove sites to your heart's content and it will not cause data to be lost, but it WILL force a restripe any time a node member of a cluster site changes so you have to be OK with that (or plan around knowing it will happen). 

 

The program will warn you (ant it does look very scarry when it does) when you switch from one site to two sites with nodes of a cluster (or the other way around).  The warning is more bark than bite as the only affect is that it is warning you that a restripe will take place and that the additional rules required for a multi-site cluter will no longer apply (or will apply if you are converting to a multi-site).

pirx4711
Frequent Advisor

Re: change site configuration - move LH nodes to different site

Resriping ist bad. Last time it took 8 days after one node failed. So if I remove 8 nodes frome the obsole sites and add 4 to each of the remaining sites _one_ restripe will happen after I've added the last node?

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

I never said it was fast just that it isn't bad (you can slow it down so it doesn't affect production and you can do it online without any data loss). 

 

EVERY time you change a node's site it will cause a restripe so you better to it all at once through the site management section or be prepared to be doing the process through next year!

 

I"m not convinced you are just looking to change node sites.  Can you post your plan of what you have and what you want?  something like:

Initial state: managment group $ cluster X with node A/B/C at site 1 and D/E/F at site 2

Final Plan:  cmanagment group $ luster X with node A/B/C/D/E/F at site 1.

 

The above change CAN be done and it will require one restripe.  Depending on the order of the nodes in the cluster already the restripe might not actually take any time at all, but I don't want to confuse the issue here and if you really want to understand what is going on you can answer EVERYTHING you asked by simply reading the manual.

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

after re-reading your OP, there is definitely a shot that if you have site A/B/C/D with cluster one at A/B and cluster 2 at C/D, a switch of cluster 2 to A/B will generate a quick restripe where data doesn't actually have to move significantly between the nodes as opposed to if you set it up as B/A where data would have to flip across the nodes, but this is all dependent on the order of the nodes and the sites on the cluster. Best thing to do if possible is post a picture of the cluster nodes w/ their order in the cluster and their sites shown.
pirx4711
Frequent Advisor

Re: change site configuration - move LH nodes to different site

It currently looks like this:

 

- one mgmt group, 20 nodes, 10 esxi hosts

 

cluster 1

site A: nodes 1-6; hosts 1-5

site B: nodes 7-12: host 6-10

 

The next 8 nodes were put into a new cluster but the same mgmt group. For whatever reason 2 new sites were created, although the new nodes are at the same location / racks as the first 12 nodes.

 

cluster 2

site A-02: nodes 13-16; no hosts

site B-02: nodes 17-20; no hosts

 

 

Because the 2 new sites does not have any hosts assigned - which I think is not right - I want to move the 4 nodes from site A-02 to A and the other 4 from  B-02 to B. So I first have to remove them at the -02 sites and the ad them to the remaining.

 

I don't like the idea of restriping. I'll post a screenshot of the sites in CMC on monday.

 

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

ok.  well it makes sense to move the nodes to the correct sites.  The ORDER of the nodes in each cluster is the critical issue in determining how a LUN gets striped across the nodes. 

 

Lets assume nodes are in the order 1,2,3,4,5,6,7,8,9,10 and when you expand the list of nodes in the cluster they start at the top w/ 1 and go down in order to 10 at the bottom.  In the case of a NR10 LUN in a single site cluster, the data will get mirrored from the odd nodes to the even nodes and the data will be striped across all odds or all evens so that all data would be stored on both an odd and an even node.  You could then lose even EVEN node or every ODD node and your data would still be available, but you could lose availability with the loss of the "right" two nodes if you lose the odd and even mirror of eachother.

 

When you get into multi-site clusters, there is a rule for the logic over the logic for data described above which makes sure that your data doesn't keep the mirror at the same site, so if you have nodes 1,2,3,4,5,6,7,8,9,10 w/ 1-5 at one site and 6-10 at the other, your data would be forced to be mirrored w/ your stripe across 1-5 which would mean that you are pretty much screwed and will most likely force a re-stripe...  The reason is that because I don't think you can bulk change the sites through CMC which means you have to remove the nodes from one site and then add them to the other and during that period when they are unassigned it MIGHT initiate a restripe (I don't know if it will or won't).

 

Not much you can do other than try removing all the nodes from site A2 and then adding them to site A and THEN repeat for B2 to B... don't remove sites A2 and B2 at the same time or you will definitely start a restripe as the nodes reorder for a single site configuration.

 

The alternative is that you need to ask support to see if they can go in and convert A2 and B2 to A and B.

pirx4711
Frequent Advisor

Re: change site configuration - move LH nodes to different site

I already contacted support, they told me that there would be no restriping. Attached is the detail view. I'm feeling uncomfortable with this change, because I don't know for sure what will happen.

 

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

if you first move the systes in RZ-cl02 to RZ without changing TC-cl02 and THEN repeat for TC-cl02 to TC you should not experiance a restripe.

 

Honestly, a restripe isn't the end of the world... it just delays your impliment of the next step (which given the length of this thread would have already been completed.  A restripe w/ all nodes in good health goes much quicker than a restripe to recover from a broken node.

 

Side note, it looks like the first site was sentup nicely with evens at one site and odds at another... it would have been better if you setup the 2nd cluster w/ that same pattern because then you could even switch between single/multi-site without a restripe.

 

If you are still uncomfortable the manually really does do a pretty good job of going over how everything works.  Read it or search the CMC help window and read up there and you should feel better.

pirx4711
Frequent Advisor

Re: change site configuration - move LH nodes to different site

Thanks for your patience. I reorganized the sites and no restriping occured. I'm extra cautious with this because we had a lot of trouble with our LH environment in the past 2 years and not everything worked as it should or I was told it should.

 

Last time a complete node failed (3 disk error) and we had to setup the node from scratch (by HP support) it was not added correct to the cluster. So resriping began and took ~10 days. Then support noticed that there still was a ghost node (the old one) and they then removed it. After that restriping did start again fpr 10 days. So we had a total of 20 days and some volume were in state unprotected.

 

 

oikjn
Honored Contributor

Re: change site configuration - move LH nodes to different site

yikes.  That sounds like a major headache...  better than dataloss, but still not comforting at all.

 

The good news w/ this situation is that if for whatever reason the cluster does decide to do a restripe, you will never lose your data redundancy so you won't be left naked like your botched node replacement.

 

As for the speed of a re-stripe, thats all case specific based on your load and the data change and continued change rate, AND the bandwidth you allow the cluster to use to do the restripe, but given that you are talking about 10-nodes, a data-restripe is likely to be non-trivial.