Is Aruba 5406R with VSF too much for redundancy?

[emsc]

Hi,

This is my first post, so hello everyone.

We have a new 3 racks setup. The idea is to have 2xToR per rack + a 5406R as distribution switch.

My question is regarding availability. Does it make sense to invest in 2x5406R and use VSF or does one 5406R with two power supply and 2 management modules enough to provide redundancy and failover?

I guess that more is better (?) but does it really make sense for a small setup like this (that should be available al the time) 

thanks!

em

[parnassus]

Interesting question.

IMHO setting up two Aruba 5406R zl2 in a VSF Stack should require a further step (at least involving any Server host eventually connected to the VSF) to unleash the full power of such type of deployment: that each connected host uses two links (2 ports Port Trunking using LACP) spanning to both VSF member Switches...this hosts<-->distribution level added redundancy will avoid that, if just one of the two distribution layer Switches goes down, you then will suffer the lost of all the hosts (solely) connected to it.

I know that is something very hard (and expensive) to achive with client hosts (single NIC equipped, two cables required to Rack per each Client) but make sense considering - at least - Servers, if any Server is eventually connected through the Distribution Switch(es) inside the Rack...well...that's more simple to do.

Having VSF and hosts connected to just one Switch only is like using the VSF "redundancy" feature at half of its real power (or just using it as a expensive way to expand "horizontally" your network, which could be the case anyway, depending on available budget): I mean, that's great...but...what will happen if/when one Switch of the VSF Stack goes down? say goodbye to its directly and solely connected hosts.

More or less that is the same problem one face when 2 or more Comware based Switches are depolyed in a IRF Stack: Is the IRF feature used to expand the Network horizontally only? or is it used to achieve real redundancy at hosts<-->distribution level too? or possibly to achieve both goals?

This may explain why, usually, the role of Aruba 5400R zl2 Switch Series is to act (or be placed) as Core switch (in both cases: when depoyed as single Switch and when deployed in a VSF Stack).

Said so...if your Distribution level numbers can be managed with just one Aruba 5406R zl2 (in terms of used ports involved, modules used and type of connected hosts) probably using just a single Aruba 5406R zl2 equipped with (A) redundant Power Supplies and (B) redundant Management Modules is the fastest and cheaper way to go, far less expensive than buying two identical Aruba 5406R zl2 Switches (consider that you need 10G or 40G ports for linking VSF members).

You should then be aware that if - in a near/far future - you are going to deploy VSF you will:

• (Limitation) lose the second Management Module (it must be put in Offline mode to avoid issues) on each VSF Member.
• (Requirement) required to provide/use only v3 zl2 Modules.
• (Requirement) required to made the Aruba 5400R zl2 working in v3-mode only (so loosing any v2 zl Module).
• (Requirement) required to use 10G or 40G ports for the VSF setup, possibly using (as a best practice) ports on different Modules.
• (Requirement) required to deploy a MAD Device (such as an Aruba 2920).

but, apart from these restrictions/limitations, you will gain in redundancy (protected by dual Power Supplies per unit too) and you will be able to expand your Network horizontally (growing it quite flawlessly).

[emsc]

Thanks for the explanation.. let me add a bit more info about my planned setup.

We will have 2 ToR on each switch.

• These ToR will be stacked; either 2xFlexFabric 5700 (with IRF) or 2xAruba3800 with stacking module.
• All servers in the racks will have 2 network cables. One cable to each ToR. We will then configure LACP on each server and Trunk/LACP on ToR.
• From each ToR we will uplink with SFP+ to our core/distribution switch(es) Aruba5406R.

Question:

•  Does this extra info help, o changes anything? :)
• If I start with a single 5406R (double power, double management) and then decide to add another 5406R, can this be done without down time (adding the second one and configuring VSF)?

Questions regarding LACP setup (can open a new Post if needed):

• Will this LACP on server side enough to expect a switch to die and continue with operation? Or do I need to also tell the ToR to LACP it's uplink?
• On the stacked ToR I configure a LACP Trunk with both ports where the 2 network cables from the server are connected. Do I also need to LACP Trunk on the 5406R (If I use two, with VSF). I believe the answer is yes, but just in case.

thanks again.

[parnassus]

Wait just a minute and watch this presentation ([ATM16] Take a Walk On The Wired Side): there is a Tier-3 scenario considered at the end but also explains the positioning of Aruba 5400R Switch Series with respect to FlexNetwork Switches.

I think VSF, when depolyed with the Auto-Join/Plug&Play procedure, doesn't require the reboot of the first Switch but for sure requires the reboot of the second one added. On the contrary when VSF is deployed using the Manual Configuration procedure a reboot of the first Switch and then a reboot of the second one is required. Using the VSF (Loose/Strict) Provisioning procedure apparently - but I could be wrong here - requires to reboot the first Switch only...even if I'm not sure (to me this procedure will require the reboot of the second one too).

So only the Auto-Join/Plug&Play procedure will grant you no downtime IF you already have an Aruba 5400R zl2 working in v3-mode only.

If your Servers will have two aggregated links that will span to two different members of the ToR that will be OK...since the ToR's Stack (made with IRF in case of FlexNetwork Switches), from the point of view of your Servers, acts as a single virtual Switch...so LACP from Servers to the IRF will be good enough.

IMHO you will need to use redundant links (LACP) between the tiers (Distributinon and Core) in both cases:

• if you are going to use Aruba 5400R zl2 as a single or in a VSF Stack.
• if you are going to use Aruba 3800 single or (Hardware) Stacked.

I think in both cases LACP links should span - as a best practice - to both members (VSF of 5400R zl2 or Hardware Stack of 3800) but, as I understood, it will not be a particular Distributed Trunking (lacp-dt)...just a plain Trunking (lacp) between tiers...since both ends (Distribution and Core), cause the stacking, act a single virtual switches.

[parnassus]

Probably, if I were you (you're a lucky guy with all that metal!), I would deploy two HPE FlexNetwork 5700 with IRF as ToR at Core and then I would use (Hardware) stacked Aruba 3800 Switches as distribution level..but...not definitely not Aruba 3800 as ToR at Core (using then those Aruba 3800 at Core with one or two Aruba 5400R zl2 at distribution level make no sense at all to me).

[emsc]

Maybe I wasn't clear about the design.

The Aruba 3800 (or FlexNetwork 5700, depending on $$) will only be for server ToR. Then they will be uplinked to 5406R via sfp+.

This uplink will end up in the 5406. 

So, at the end, the 5406 will only have the uplinks of the ToR. 

[Ian Vaughan]

Howdy,

What I would be asking is -

             What is the underlying requirement in this "mini-datacentre" build?

A - Applications

B - Infrastructure building blocks

C - Connectivity 

D - dependencies 

How many hosts / ports / storage?

What do the traffic volumes and application spread look like today and how are they expected to ramp up?

Do you just want a Layer2 fabric with routing in the "core/aggregation" layer?

     You could just IRF all of the ToR switches together in a big ring...

Do you want a fully meshed Layer 3 routed Spine-leaf like topology?

       Very popular these days Sir. Routing right down to the edge, multiple uplinks, equal cost multi-path OSPF etc. 

Does this network need to participate with other networks either in other premises or with virtual infrastructure in a public cloud somewhere?

Will you be running Virtual switches in the servers (that we haven't yet mentioned) in the racks? 

Should your Server guy be looking at less servers but with 25Gb NICs plugged straight into a bigger 2 switch Core and forget about ToR?

Do we need to think about any overlay networks that our compute minded colleagues might need to achieve some layer 2 stretchiness either between racks or between premises? 

Are there security appliances (real or virtual) and or load balancing appliances (again real or virtual) that we need to factor into our design?

Well,

Firstly, I'd look at 5700 in IRF pairs in the racks with a pair of 5940 sat over them. 40Gb Bidi conenctions to keep the cost down - loads of active-active 40Gb non-blocking. Think about including a segregated 1Gb network (can just be a single switch) just to plug the management ports into and keep them off the "prod" LAN.

The (formerly procurve) aruba-OS switches are great in the Campus but IMHO Comware* having IRF, mBGP, VRF capability, finegrained-QoS, storage networking, eVPN, VXLAN VTEP, QinQ, deep debug and front / back airflow makes them the clear winner in a datacentre / server connecting type role. 

* - Features vary by model etc :-) talking at least 5700/59x0 

Paint us a picture, give us a use case and we'll help best we can.

I am a fan of getting the footprint as small / lean as possible (max perf per watt or max perf per 1U of rack) and then using the least number of biggest bandwidth per $ links to lash it together. You might be able to do it all with 2 good switches :-) 

Hope that was useful - plenty to get stuck into.

Ian

[emsc]

Hi Ian,

Here is more info.. hope it helps.

I'll try to be as specific as possible, without going out of the subject.

Main Idea: 

• 3 racks (initially), with 1 for staging (not important) and 2 for production.
• Everything needs to be redundant, so if one racks goes down, we are still online.
• We have two internet lines dropped from our ISP (active/passive with bgp). One line goes to Rack #1, the other to Rack #2.

Each production racks (two) will have:

• Firewall (active/passive with heartbeat)
• load balalncer
• virtualisation servers

Future: Should be easy to grow horizontally at the virtualisation level. Future racks will have a ToR and servers. 

Our current idea:

• ToR on each rack (5700 or 3800 - depending on $$), 2 switches stacked, with LACP on servers, connected with 1Gb network
• Each ToR will uplink (SFP+) to our core/aggregation (one or two) switch(es) (5406R?) 
• Firewall will also go directly to core/aggregation
• Dropped line from ISP will also go directly to core/aggregation

Questions:

• Does this design make sense?
• Will this give us redundancy at all (network) levels?
• Should we get one or two core/aggregation switches?

Notes:

• We are not expecting a huge amount of data in the beginning, so 40Gb seems a bit too much.
• Traffic will come to the firewall which will then pass most of it to the load balancer.
• Firewall will also be the default gw of all servers (on all vlans)
• We will virtualise, but not use any virtual switch or so. Only Vlan tags on the server side.
• This network will participate in other networks, but via VPN tunnel (not sure if relevant)
• We are looking at more o less 10 physical servers per rack
• Simple is better. I prefer a easy to manage setup, since we will be dealing with it, and none of us is a network expert :)
• Price do matter, but also quality. (so, not too expensive and not the cheapest)
• Storage is not included here because it will have a separate network layer.

Thanks for the help!

[parnassus]

Hi emcs,

@just a clarification, you wrote:

---

@emsc wrote:

Main Idea: 

• 3 racks (initially), with 1 for staging (not important) and 2 for production.
• Everything needs to be redundant, so if one racks goes down, we are still online.
• We have two internet lines dropped from our ISP (active/passive with bgp). One line goes to Rack #1, the other to Rack #2.

Each production racks (two) will have:

• Firewall (active/passive with heartbeat)
• load balalncer
• virtualisation servers

Our current idea:

• ToR on each rack (5700 or 3800 - depending on $$), 2 switches stacked, with LACP on servers, connected with 1Gb network
• Each ToR will uplink (SFP+) to our core/aggregation (one or two) switch(es) (5406R?) 
• Firewall will also go directly to core/aggregation
• Dropped line from ISP will also go directly to core/aggregation

---

Since you wrote "Each production racks (two) will have:"...does this mean that you're going to have one "HA Firewalls cluster" and Load Balancer per Rack (so two "HA Firewall Clusters" and two Load Balancers in total) or that you're going to have the Active Firewall on Production Rack 1 and the Passive Firewall on Production Rack 2 and just a single Load Balancer shared for both Racks?

If the right answer is a Go for the first case (Two "HA Firewall clusters" and two Load Balancers in total) this means that you're going to have two Active+Passive ISP Links per each Production Rack...is that right?

One answer or another, I imagine the (or each) HA Firewall Cluster (one unit Active, the other Passive/Standby) is going to be depolyed using VRRP, so all the network's hosts will be connected to the (virtual) Default Gateway IP Address provided by that HA Firewall Cluster.

If so your ISP will drop an Active and a Backup/Passive connectivity (2 links) on each Production Rack for each HA Firewall Cluster...so you're going to have a grand total of two Active and two Passive. Strange but totally possible (will be then interesting to see how you manage the second (virtual) Default Gateway IP Address provided by the second HA Firewall Cluster with respect to your Network hosts...something like a secondary/alternative virtual Default Gateway if the first isn't available to manage outgoing traffic).

Possibly here I'm totally wrong but all depends by that first sentence "Each production racks (two) will have:".

Clarification closed.

From what you wrote it seems that all edge connectivity (physical/virtual Default Gateway(s)) is going to be connected to the Distribution layer...and not to the Core one. If so you need to provide redundancy to the Distribution Layer...and - considering that Distribution layer will have clients only - the best is to go with Aruba 5406R deployed with dual Management Module and Redundant Power Supplies, if you are for the ArubaOS Switch on Distribution side.

It's interesting to note that, IMHO, you're doing the contrary of one would expect (it's hard to know if your important traffic should be the one destined to Servers or the one that is concurrently destined to Servers and also coming from Clients): the pyramid up-side-down, where the distribution layer has less units (at worst one, at best two in VSF) than the core (where you will have two ToR Switches deployed in IRF if Comware based or deployed with Hardware Stacking, if ProVsion based).

Again maybe I'm wrong and I totally misunderstood the whole picture...but it's what I'm able to figure reading your described setup.

Why not considering an all-Comware based setup that will span through both Racks deployed with a two full-Ring topologies (with MAD Device)...one ring for ToR and one for Distribution (with less expensive Switches than the HPE FlexNetwork 5700)?

[Stephen A Swain]

Questions:

• Does this design make sense?
• Will this give us redundancy at all (network) levels?
• Should we get one or two core/aggregation switches?

In answer to the "redundancy at all (network) levels", I'd suggest consider having two dist/core switches. Here's why...

If you have a 5406R with redundant management modules for instance, there are some cases where downtime is required to perform maintenance functions, and there is only a single fan tray. There is only one software manager active, and even with all of the non-stop redundancy options configured properly, there are some cases where downtime can happen. For example software bugs, power failover issues (the power redundancy is not 100% uptime, power failures can cause small outages for components in the chassis).

I had a situation once where the fan controller in a 8206zl (older model, similar design) got wet, the fan went down, and the switch shutdown. In the 5406R there is more redundancy than the 8206zl, but still a single fan tray.

With two 5406R with VSF, 3810M Stacked Chassis, or Comware IRF at core/dist, you get a higher uptime due to every component being redundant, even the underlying hardware and operating system. Two physical systems in a virtual chassis will normally have higher uptime, but it just costs more. Chassis internal redundancy is great for the budget though.

I'd also be checking the guides and manuals for each model choice on how upgrades are performed, and what kind of downtime is present. At some point you will need downtime to make changes and upgrades. Some virtual switch designs have less downtime for management than others.

Btw, many DC fabric designs these days are moving away from traditional hierarchical designs and moving to spine and leaf mesh designs. The redundancy is built using TRILL/SPB or similar and the software is modular, or allows in service software upgrades (issu). These designs allow a higher uptime and may be a better match for a DC build. Sometimes DC designs are better served by flat mesh designs (flattened networks).