Link Aggregation/ Trunking

[mjwlaz]

I am very new to the idea of link aggregation and am hoping one of the experts here can help me use it to my advantage.



First some facts:

I have 2 Baseline 2848-SFP Plus Switches

I have an intensive SQL application that comprises about 70% of the software our company relies on for doing business

I have a couple of Broadcom NetExtreme NICs configured for "Smart Load Balancing"



Smart Load Balance and Failover In this type of team, a standby member handles the traffic if all of the load balance members fail (a failover event). All load balance members have to fail before the standby member takes over. When one or more of the load balance members is restored (fallback), the restored team member(s) resumes the handling of the traffic. The LiveLink feature is supported only for this type of team.



I was thinking about teaming when I took a look at my switches. I'm pretty sure that with Broadcom's Smart Load Balance and Failover configuration my NICs team up to increase the throughput to the server AND one backs the other up in the event of failure.



Is there a benefit to creating a "team" in the switch as well?



If I group the ports on the switch what does the combined throughput become?



If I group 2 ports on the switch where is the combined throughput directed?



What I'm thinking of doing is this:

Switch A: Create a 2 port group. One of the ports will be connected to NIC A in my server.

Switch B: Sames as above except connected to NIC B

On the server NICs A & B will remain teamed.



Will this work? What are the specs of the improvement I will see?



If 2 ports on the switch are grouped are they then allowing 2GB to pass through instead of the 1GB each one is rated for?



If so, does it follow that the server will then be getting 4GB of data passed to it as a result of the grouping on the switch?



If so this creates a hefty bottleneck at the NIC doesn't it? When 2 1GB NICs are teamed are they capable of handling 2GB of data? Does it make sense to pass them 4GB of data if they can only handle 2GB?



I'm ultimately trying to get as much data as possible to and from this server while maintaining failover at the NIC and Switch levels. Perhaps what I need to do can be explained more simply.



I've got a bunch of questions so before I go too much further I'm going to wait to see what my responses are. Thank you very much for helping me understand and work this out. I'm a one man show in a small company and this area is not my specialty.



MJ

[Fred_Mancen_1]

Hi.



At first, let's consider just the switches, ok? These Baseline switches are stand-alone models, I mean, they have separate functions when working. So, if you are trying to group 2 links (one link per switch) this would not work. This feature makes sense if you have stacked switches (models 4500, 5500-EI, 5500G-EI, 4800G), that supports distributed link aggregation.



Your server probably supports LACP, that will be the protocol used in this feature. In a failover scenario, nothing will change if you group the switch ports where the server is connected.



To increase the throughput and resilience, you need to create a manual link-aggregation group in a switch stack, distributing the links between the units in the stack. The throughput will be 2Gb (4Gbps in full-duplex mode), and if one link fails, the other link maintains its work normally, in a transparent way.



Your questions can be resumed in a single concerning: if you want to increase bandwidth and provide contingency using your Baseline switches you need to create a manual link-aggregation group in just one of your Baseline switches, using both links connected straight to this switch, using LACP on the server also (teaming).



In the other hand, if you want to distribute the links between two switches, you'll need to use stackable switches, not the Baseline. In this scenario, using Baseline switches, probably you'll face problems in the core when receiving the same MAC Addresses coming from two different paths.



Hope this helps.

[mjwlaz]

Thanks for the reply Fred. I have a couple of more questions if I may.



I'm curious about your comment about using stacked switches vs. the Baseline switches. Did I misspeak and lead you to believe that I wanted to create a group that spans two switches or did I misunderstand what you are saying.



My hope is to connect NIC A to Switch A and create a group in Switch A that is made up of the port NIC A is connected to and an empty port.



I would then connect NIC B to Switch B and create a group in Switch B that is made up of the port NIC B is connected to and an empty port.



If I understand your reply correctly you addressed a scenario where the port that NIC A is connected to on Switch A and the port that NIC B is connected to on switch B are grouped. That's not my intention. I wonder though if you got that impression because grouping a connected port and an empty port on the same switch won't yield the results that I am looking for. Is this the case? My naivety may be the problem here.



As I'm rereading your post I believe that the little bit of knowledge I have may be the culprit. I definitely don't need to create a group between ports on two different switches. I am looking for the aggregate throughput of 2 ports on 1 switch where only one of the ports is connected back to the server. Having written that I suspect that in order for me to get these results both ports on one switch need to be populated and then grouped. Right?



Another question, if I create a group in the switch why do I also need to team the NICs? Don't misunderstand me, I plan on teaming the NICS, I just want to know why I need to if a group is created in the switch.



I think for my purposes I need to put in a 4 port NIC and connect 2 ports to Switch A and group them, and 2 ports to Switch B and group them to accomplish what I want. This will allow me to create a successful group on each switch. The NICs are split between two switches as part of our emergency plan. If one NIC fails, the other picks up. If one switch fails the other picks up . . . all the way down to the circuit breaker the components are connected to.



Some other thoughts:



If I do as I mentioned and install a 4 port ethernet card, and then create a group on each switch I will effectively sending the aggregate throughput of 4 ports to my NIC. That's 4GB (8GB full-duplex then?, right?.



Thanks for keeping up with me. I think after I ask a question or two of Broadcom I might have this thing down.



MJ

[Fred_Mancen_1]

Ok, buddy...sorry for my misunderstand.



So, if you connect a server with 4 NICs in this scenario (2 NICs connected to switch A and 2 NICs connected to switch B, with two groups, one per switch), the only problem I see is regarding the MAC Address, that would be announced to the core switch coming from two different paths. That's why I recommended the use of stackable switches; Baseline switches are stand-alone, and they are connected to the core, right?



So, if you have a IP and MAC address coming from the switch A and the same IP and MAC address coming from the switch B, the core switch will detect that a duplicate address is coming from two different paths, and it will force a re-learning in the ARP table, consuming CPU and memory.



Regarding the throughput, actually you'll not increase the throughput to 8Gbps, because just two NICs are grouped per switch (2Gbps each group, 4Gbps in full-duplex), and at the same time causing ARP problems to the core switch. You'll obtain a throughput of 8Gbps if all the NICs (4) are grouped in the same group and switch.



I'm not sure if I was clear, but when I say that you'll need to team the NICs, that's why when it working in failover mode the server does not perform load balance using both NICs at the same time, but using just one and activating the hot standby NIC if the primary fails. This way, the server is using just 1Gbps to communicate with the LAN.



Regards

[mjwlaz]

Fred - thanks again for staying with me. Let's say I connect both of my existing NICs to Switch A and create a group, and then team the NICs using Broadcom's software. If the switch is allowing 2Gbps of throughput but the NIC team can only handle 1Gbps then is their a point in grouping ports on the switch?



For the sake of simplicity I'm going to use my current config for this next question. As I have things set now the physical NIC A has it's own IP and MAC address. Physical NIC B has it's own IP and MAC address. The team that both NICs are part of has it's own IP and MAC address. The network components identify the server that these NICS are in by the team address. How does each switch identify them? Where can I look in the switch to see what IP address/ MAC address each switch associates with the NIC that is connected to it? I'm starting to wonder if my existing team could already be causing the problems you mention with using two standalone switches.



If each switch sees the IP and MAC address of the physical NIC and not the team then I can use a 4 port NIC, make 3 teams (2 trunking and 1 failover) and accomplish everything I want to. Of course there is one more thing I can't seem to reconcile:



If I have a gigabit LAN, do I gain anything from doubling or quadrupling the amount of data that is passed through the switch to the server?



One last comment: "core switch". Hmmmm. These two Baseline Plus switches are the only two on our network. Up until now I haven't even thought of using any network management features. I don't know enough to do it. I'm actually interested in trying to figure out how I could use a VLAN or why I would use it so perhaps I can get a little more out of my switches but I think if I explore that before I get a grasp on link aggregation my mind might shut down.



Thanks yet again for helping me work through this.



MJ



Quick Edit: How does this affect my situation: BASP teaming modes BASP teaming modes include the following: Smart Load Balancing. This proprietary Broadcom technology provides fault tolerance and load balancing based on IP flow. This feature can balance IP traffic across as many as eight team members for both outbound and inbound traffic. In this mode, all adapters in the team have separate MAC addresses. Smart Load Balancingâ„¢ (SLB) provides automatic fault detection and dynamic failover to another team member or to a hot-standby member, and works with any switch or hub.

This message was edited by mjwlaz on 11-5-09 @ 8:29 AM

[Fred_Mancen_1]

Q.: Let's say I connect both of my existing NICs to Switch A and create a group, and then team the NICs using Broadcom's software. If the switch is allowing 2Gbps of throughput but the NIC team can only handle 1Gbps then is their a point in grouping ports on the switch?



A: No, using this scenario you'll have a 2 Gbps throughput, since the ports is grouped into one link-aggregation group in the same switch.



Q: As I have things set now the physical NIC A has it's own IP and MAC address. Physical NIC B has it's own IP and MAC address. The team that both NICs are part of has it's own IP and MAC address. The network components identify the server that these NICS are in by the team address. How does each switch identify them?



A: The switch recognizes the virtual IP and MAC address generated by the team; the real MAC and IP addresses are ignored by the switch.



Q: Where can I look in the switch to see what IP address/ MAC address each switch associates with the NIC that is connected to it?



A: You can see the MAC address announced in the ports which the NICs are connected, it will be the virtual MAC. In the ARP table, this MAC will be associated to the virtual IP address.



Q: I'm starting to wonder if my existing team could already be causing the problems you mention with using two standalone switches.



A: I don't think so; since you tell me that you don't have a core switch. But, in the other hand, if these Baseline switches are connected between itselves, maybe you could see that there's some warning message regarding a duplicated IP address. I said maybe...I don't know how is your real network topology.



Q: If each switch sees the IP and MAC address of the physical NIC and not the team then I can use a 4 port NIC, make 3 teams (2 trunking and 1 failover) and accomplish everything I want to?



A: No. Actually this feature does not work as you thinking. When you create a manual link-aggregation group (in servers connections the LAG must be manually created), it uses the LACP protocol to signalling between both peers - the switch and the server. Since the switch sees just the virtual addresses, and can handle just one source IP in the LAG, you cannot setup your connection as you mentioned.



Q: These two Baseline Plus switches are the only two on our network. Up until now I haven't even thought of using any network management features. I don't know enough to do it. I'm actually interested in trying to figure out how I could use a VLAN or why I would use it so perhaps I can get a little more out of my switches but I think if I explore that before I get a grasp on link aggregation my mind might shut down.



A: If you have just these two switches in your network, and I wonder that these switches are interconnected, you cannot create a LAG in one switch, another LAG in the second switch, and expects that the server will have the sum of both connections...If you want to use VLANs in your network and if you need that these VLANs could communicate between it, you will need a Layer 3 device (a router with 802.1Q support or a multilayer switch), that will route the traffic among the VLANs.



I will try to resume (and sorry if I'm not quite clear) the best way to provide 8Gbps of throughput using your 4 server NICs:



Create a manual link-aggregation group (LAG) in one of the switches, group 4 ports into this LAG, connect the server to these 4 ports as teaming, and then you'll have 8Gbps of total throughput. You'll never get 8 Gbps of throughput using two LAGs, one in each switch. This way you'll create an ARP resolution problem. To solve the problem of bottleneck between the switches, you just need to enable LACP on two ports in each switch, and use these ports to create a dynamic LAG to the uplink. But be careful: the ports that belong to a LAG must have exactly the same configuration. Whatever the LAG mode, the ports always must have the same settings.



Difference between failover and Aggregated Link:



Failover: just one NIC active at a time. The second cames up only if the primary fails.



Aggregated Link: both NICs are active at the same time, if one fails, the connection remains active and doesn't cause impacts on the network. Additionally, it aggregates the speed of the ports according the number of ports grouped in the same LAG.



Q: How does this affect my situation: BASP teaming modes BASP teaming modes include the following: Smart Load Balancing. This proprietary Broadcom technology provides fault tolerance and load balancing based on IP flow. This feature can balance IP traffic across as many as eight team members for both outbound and inbound traffic. In this mode, all adapters in the team have separate MAC addresses. Smart Load Balancingâ„¢ (SLB) provides automatic fault detection and dynamic failover to another team member or to a hot-standby member, and works with any switch or hub.



A: I'm not sure, but if I understand this feature, the Broadcom NICs just works using a proprietary feature (SLB) or in failover mode. Do you know if these NICs does support LACP (IEEE 802.3ad)?



HTH