Storage Boards Cleanup
To make it easier to find information about HPE Storage products and solutions, we are doing spring cleaning. This includes consolidation of some older boards, and a simpler structure that more accurately reflects how people use HPE Storage.
HPE StoreVirtual Storage / LeftHand
cancel
Showing results for 
Search instead for 
Did you mean: 

Multiple Failover Managers?

SOLVED
Go to solution
DSmith1
Occasional Visitor

Multiple Failover Managers?

Hi,

 

I'm about to install a system which comprises 2x 4530 StoreVirtual nodes, and 3 vSphere servers.  It's all going to be in one rack, but that aside, High Availability is the name of the game - I need to be able to survive failure of a SAN node, or a server, (or preferably both) without anyone noticing more than a minor glitch (the odd vSphere HA recovery I can cope with).

 

I was planning on deploying a Failover Manager on one of the vSphere servers on local storage, joining it to the StoreVirtual cluster, and then powering it off and cloning it (IP address, MAC address and all) to the local storage on the other two vSphere servers, as a backup plan, and then just have one of them manually powered on at any given time.

 

Then, it struck me (and I've not been able to find the answer to this anywhere I've looked, and I don't have a test system to hand I can "play" with) - is the StoreVirtual cluster limited to a single Failover Manager?  Could I just deploy three Failover Managers, one on local storage of each server, and have them all running, live, in the cluster - meaning I wouldn't have to manually "move" the Failover Manager around if the server it was running on failed?

 

I know it's not an ideal situation, and before anyone helpfully points out that the Failover Manager would be better off in a separate location or server, I just don't have another server on which to run the Failover Manager, or the network infrastructure to run it elsewhere .

 

Thanks

12 REPLIES
Bart_Heungens
Honored Contributor
Solution

Re: Multiple Failover Managers?

Hi,

 

It is rather simple: 1 single FOM possible...

 

Know that you can move/reboot/whatever your FOM as much as you want, as long as your StoreVirtual nodes are up and running, you can do whatever you want with the FOM. It is just when there is a network issue between the 2 nodes that the FOM will create quorum/majority and will decide which of the 2 nodes that will remain having active iSCSI sessions...

 

 

Kr,

Bart

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
DSmith1
Occasional Visitor

Re: Multiple Failover Managers?

Thanks for your reply - I'll stick with the cloned "single instance at a time" manually managed FOM then.  I would hope the liklihood that I lose both a server and a SAN node simultaneously - and everything else stays up - given everything's on the same redundant power feeds is a scenario that's sufficiently unlikely to happen that I can cope with a manual solution.

Bart_Heungens
Honored Contributor

Re: Multiple Failover Managers?

I know the feeling. The placement and location of the FOM is most of the times the biggest 'challenge'. Which it shouldn't be... Most of the time it is placed (at least in my case) on a seperate small/cheap/little server outside the server farm (even sometimes on workstations or the HP MicroServer or so) which acts as management server or physical DC with a Hyper-V role enabled or VMware workstation or so... Plenty of choice, but not everyone likes too much choice...

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
DSmith1
Occasional Visitor

Re: Multiple Failover Managers?

The problem I have is that the storage fabric is dedicated 10Gb, and isn't routed outside that rack.  And to route it outside the rack = $$$$$.

 

If only there was a way to utilise the 1Gb NICs, and have the FOM running on a totally separate network, that would be great, but seemingly impossiible.  Hence having to run the FOM on one of the servers that has the 10Gb NICs that actually uses the storage...

Bart_Heungens
Honored Contributor

Re: Multiple Failover Managers?

Know that the FOM does not need 10Gb connectivity... It even works on 100MB. Also the 2ms latency is not required... Since the FOM is just configuration management trafic between the managers in the systems, no data trafic will be involved. And so no need for 10Gb... But yes it needs to be on the dedicated iSCSI network...

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
DSmith1
Occasional Visitor

Re: Multiple Failover Managers?

Yeah - and that's the issue, I've got no (cost free) way to bridge my 10Gb fabric to a 1Gb fabric - so there's got to be a compromise...

Bart_Heungens
Honored Contributor

Re: Multiple Failover Managers?

Mmm will be a 'challenge'...

Place it on one of the hosts and move it around during maintenance windows... Not ideal but good to start with...

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
DSmith1
Occasional Visitor

Re: Multiple Failover Managers?

That's the reason I want to keep a clone on each server after I've added it to the cluster - no "moving about", just power one off, power another one on.  I trialled that a while back and it seemed to work seamlessly, i.e. the physical nodes had no idea it was a different VM running on a different server.

 

You don't know of anything that might cause that to break after a certain duration, that would make it a really bad idea?  e.g. you can't do the same thing with a Windows domain-joined computer, because they change their computer password periodically, so the clone would end up out-of-date with no domain trust relationship when it was powered on.

Bart_Heungens
Honored Contributor

Re: Multiple Failover Managers?

At least I never tried this scenario before... Could work, however if the FOM on the second or third server is too long offline it might have older information on the cluster compared to the physical nodes... Every time you create, change, delete a volume or create snapshots there is exchange of information between the managers...

Could work, but don't feel confident about it... And probably not supported by HP...

What about VMware FT?

--------------------------------------------------------------------------------
If my post was useful, clik on my KUDOS! "White Star" !
My blog: http://blog.bitcon.be
oikjn
Honored Contributor

Re: Multiple Failover Managers?

I haven't tried the clone approach, but I would stay away from the clone idea.  It takes all of 3 minutes to migrate the FOM around so it will be no problem to just move it when you need to reboot its host.

 

We put our FOM on our DPM server.  We are a hyper-v shop and setup our DPM server as a VM on a dedicated host which just runs the DPM VM and the FOM and its been working well for us...  honestly, it might be "risky", but we just reboot the host with the FOM when needed as long as the other nodes on the SAN are up it just doesn't matter, but it would make for a slightly extended RTO time, but something we are OK with in our situation.  If we needed 99.9999999% then I would move it to another host so it never is at risk.

 

Avoid the clone because the stored data on it will be out of date and might end up being be more harm than good if you ever try and run it.

HPstorageTom
HPE Pro

Re: Multiple Failover Managers?

Stay away from the clone approach ... it will not really work properly. The FOM keeps a current copy of the internal management database. Starting cloned FOMs some time after the clone process only makes sure that you will get your management group into a mess by running managers that do not have the same information.

GlenRB
Advisor

Re: Multiple Failover Managers?

I see another issue here as well so someone tell me if I'm missing something.  You can have one FOM per management group.  So if the server that hosts the FOM becomes unavailable so does the FOM.  You will not be able to remove the existing FOM if you cannot communicate with it, but you will also not be able to add another one since the management group still knows about it.  It's a bit of a catch 22.  You can't remove a FOM you can't talk to but it will still be detected in the management group and give an error if you try to add one.

 

From my experience, I believe this will require a call to HP support to have them manually remove the orphaned FOM before you will be able to add another.