Managers - Confused!

[Paul Hutchings]

I have a single Management Group with two sites.

I have two multi-site clusters.

Cluster1 has 2 SAS nodes.

Cluster2 has 4 MDL nodes.

The nodes are assigned to the sites and there is an equal number in each site.

Right now I'm running 2 managers on nodes 1/2 in Cluster1 which is because when we configured the initial management group we added 2 nodes to create Cluster1 and the wizard setup the nodes.

The Best Practise Wizard is suggesting that now I have 6 nodes total I should have 4 managers and a FOM (the FOM will be added on the local storage of our primary site's vSphere server later).

What the Best Practise Wizard isn't telling me, is which of the 4 new (or 6 total) nodes I should be running the 4 managers on, and I'm struggling to make it out from the documentation.

Which should I be running them on and why please?

Thanks,
Paul

[Jim Ruzauskas · Accepted solution]

Yep, I had the same confusion on this as well. All said and done, you need 1 FOM per management group and then all nodes run a standard manager. I think you have to delete the Virtual Manager before it will let you add the FOM.

[Paul Hutchings]

Thanks Jim.

Where the documentation falls down is making that clear I think.

Should I really be running 6 managers though? I ask as the CMC best practise analyser suggests 4 (+ FOM) yet the documentation suggests (p148-149 of c02008676.pdf) that if you create a management group with "3 or more" nodes it will only configure 3 managers - I thought the whole point of managers was you need an even number in each site + a virtual manager or FOM to cast the deciding vote?

Also is manager status retained after a node restarts?

[teledata]

I always understood it to be that you have to have an odd number of total managers, with a maximum of 5.

Not sure why there is the max of 5, unless perhaps it gets to be too much metadata to efficiently replicate beyond 5 managers...

The problem I always ran into, was that you could not obtain node level SNMP performance metrics unless that node was running a manager, although I haven't seen if this issue is different in SAN/iQ 9.0

[Paul Hutchings]

I think where I've been getting confused is that whilst testing I only had 2 nodes so quorum/managers was easy - 2 physical + FOM/Virtual.

Now I have six nodes and (minimum) of two clusters I think I made the mistake of thinking that managers/quorum applied to the *clusters* rather than to the *management group*.

So as per c02063195.pdf, on page 11 Figure 3, I'd use 4 managers, 2 per site.

In theory the managers can be random nodes, but as I currently have 2 clusters, I'd make one node per cluster per site (1 node x 2 clusters x 2 sites = 4 managers) + FOM.

Does that sound about right on the misunderstanding and the solution?

[Mark...]

Hi,
In a management group it does not matter how many clusters you have as long as you have the correct amount of manager processes running to maintain quorum, because this is for the whole management group.
In your situation with 6 physical nodes you could run with 5 node managers and have NO FOM and that would be fine. If one manager node were to fail then you could still turn on the manager on the node that was not running a manager to take you back up to 5.
I have always been told to only have a max of 5 managers in a management group as well. I believe it is to do with the management overhead as well as a previous person has mentioned.
Therefore in your current situation you could run:
5 managers
4 managers + FOM
4 managers + VM
Again it has been mentioned that you are not able to have a FOM and a VM in a management group.
As you have also said, it would make sense to split your managers across both sites and then to decided which site would be your primary site to have the 5th manager/FOM. HP do recommend that FOM should be on site "C" for resiliance.
Remember you only start VM when you lose quorum so this would give you the flexibility to decide which site you would like to remain running but it is a manual process.
Mark...

[Paul Hutchings]

Yeah I've realised that's my schoolboy error :)

So in my situation seems to call for Site A designated "Primary" in the CMC with:

Site A, Cluster 1 - 1 manager
Site A, Cluster 2 - 1 manager
Site A, FOM (on local VMFS)

Site B, Cluster 1 - 1 manager
Site B, Cluster 2 - 1 manager

What would happen to if I lost the FOM though, or just needed to restart the system it's running on?

[Mark...]

Hi,
Loss of FOM or a node of two would still not be a problem as long as you maintain Q.
Example- 4 man + fom = 5 = Q of 3
Loss of FOM = 4, Q still = 3 - stays up just restart FOM when able.
Loss of node = 4, Q still =3 - so stays up again.
In both cases, being as you have at least 2 nodes not running manager process then there is nothing to stop you starting manager on one of those nodes to maintain your 5 votes.
Once you have fixed the problem then you can just stop / start managers as required
Hope that helps.
Mark...

[Paul Hutchings]

Thanks Mark, I think I'm there now - like the flexibility of the P4000, not so keen on all the "what if's" it raises :)

Toying with one of those cheap HP Microserver's to run the FOM on rather than have it on the main vSphere host in that location as it could also double up as a router/firewall between the iSCSI VLAN and the production LAN.

[Mark...]

Hi Paul,
Any PC running XP will do as you can create a FOM on there if you wish. You just need to run it under the ESX player or whatever it is!
Regards,
Mark...

[Paul Hutchings]

Thanks Mark, I know you can use basic player or whatever they call it now for the FOM, I was just thinking out loud I guess over whether there is any point running it somewhere other than on local storage on one of your vSphere hosts.

[Paul Hutchings]

Incidentally, if I had a FOM in Site A, would I still bother to designate Site A as "Primary" in the CMC?

I'm thinking what if I temporarily lose Site A and wanted the P4000 in Site B to take quorum but not permanently as the manual suggests the "Recover Quorum" command would do.

[teledata]

Since you have additional nodes available, I would not bother with the FOM, but just run the 5th manager on one of your other nodes at the primary location...

FOM was introduced to solve the original problem of requiring a customer to buy at least 3 nodes to create a high availability cluster.

Since you already have additional SAN/iQ nodes available in your management group, there isn't any benefit (that I can see) to using FOM over one of your existing nodes.

[Paul Hutchings]

"Since you have additional nodes available, I would not bother with the FOM, but just run the 5th manager on one of your other nodes at the primary location..."

Yes I was thinking about this and I'm less and less keen on the FOM as it seems to assume my primary site will always be available, and whilst it makes a link failure easy to deal with, it seems it makes a loss of Site A tricky to deal with.

Would I be correct in thinking the following is sensible?

Run regular 3 managers in Site A, and 2 regular managers in site B.

Link dies - A has Quorum.

B dies, A has Quorum.

A dies, split brain so bring up a VM in site B to regain quorum.

[teledata]

You've got it... Or (as in your other recent post) you can do your 3rd site to get the best of all worlds...

The only thing that bothers me about the 3rd site, is if site A looses all network connectivity you are down..

I would keep an offline mirror of your FOM that normally is running at the 3rd site, so if you loose all links you can launch the "copy" of your FOM to regain SITE A functionality if needed.

[Paul Hutchings]

Yeah I did think about taking a snapshot/cloning the FOM but I didn't know if it would work - I figured if I tried it during a testing period it may not reflect any "Get lost I don't know who you are" that might occur a year down the line if I suddenly bring up a cloned FOM.

Tbh even ignoring the FOM issue, I'll sleep much better with a third switch and redundant network topology - really I think I stuffed up not planning that from the outset but those Proliant Microserver's are cheaper than a PC to just run the FOM.

As for HP's SFP/GBIC pricing however......

[Terry.giblin]

Quorum:

At least 3 or 5 'nodes' to create a high availability cluster.

That's what it said on the box, in my garage.

If only, it was that simple.

30 years of VMS.

[teledata]

Since you are running your FOM as a virtual machine, you could pretty easily use VCB or periodically shut it down, and take an updated snapshot or copy of the FOM disk files...

You could even automate it with a script ie:

Sunday 3am, shutdown FOM
Copy disk files over WAN
restart FOM

[Paul Hutchings]

Yeah I may look into that.

Presumably if you lose all three sites in different order due to, say, power or "something bad", but you they all come back (at different times as power is restored) quorum takes care of taking the cluster offline, then when things come back there's a resync/quorum is restored?

I can only test so much with yanking cables...

[Terry.giblin]

"I can only test so much with yanking cables..."

How long can an 'ethernet cable' be?