NIC Bond setup - which option is best

[Keith Martin_3]

I have three p4300 nodes in a cluster, two are old, been in for over 2 years, and one is brand new.  All are G2.

I added the new one in today and the restripe started.   I noticed a best practice warning in the cluster regarding the network bond setup.

Apparently when I setup the two old nodes I chose 802.3ad for the bond type, and on the new one I went with ALB.  Seems ALB is recommended option.

What is required if you want to use 802.3ad?  Also, what is required to change the bond type, just break the bond and recreate it?

All nodes are in a stack of Cisco Catalyst 3750 switches. (3 switches acting like 1).

There are no ether-channels setup for the ports dedicated to the p4300 nodes.

p4300's are all running SAN/IQ 9.5.

I was going to change the older nodes to use ALB, but have to wait for downtime to do that.

If anyone has any input on this I would appreciate it.

Thanks,

[ccavanna]

I just put in 4 4500 nodes and did testing between LACP and ALB. I found that ALB's throughput was about 30MB/s faster than the LACP or 802.3ad configuration.

[Keith Martin_3]

ccavanna, thanks for the info.  That helps.

Now, does anyone know of a good way to make this change?  I would assume it's breaking the bond and recreating the bond.  Just wondering what that would do to communications.

Seems that it should be OK, and maybe a blip whenever I change the node hosting the VIP. 

Any other thoughts?

[CRiiser]

Hey!

From: VMware and HP Bestpractise

You can see that ALB is fine but LACP, if your network design can support it, is giving you better write performance.

So, if you can, go LACP.

/Christian

[Emilo]

The best bonding option is to leverage Link Aggregation Control
Protocol (LACP) 802.3AD if the switching infrastructure supports it. From the storage node, LACP
bonding supports both sending and receiving data from both adapters. Network teams on vSphere

This information is not correct.

Saniq 9.x uses Linux balance-alb or 6  Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation.

So it recieves and transmits on both NIC's.

if you are using ALB now run diagnostics and download the ifconfighist.log and you will see that it balance both recieve an d transmit.

I used to be advocate of link agg but after seeing the numbers for myself and the additional work in setting up the trunk group I am not so sure if it is worth the trouble.

[CRiiser]

---

@Emilo wrote:

The best bonding option is to leverage Link Aggregation Control
Protocol (LACP) 802.3AD if the switching infrastructure supports it. From the storage node, LACP
bonding supports both sending and receiving data from both adapters. Network teams on vSphere

This information is not correct.

Saniq 9.x uses Linux balance-alb or 6  Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation.

So it recieves and transmits on both NIC's.

if you are using ALB now run diagnostics and download the ifconfighist.log and you will see that it balance both recieve an d transmit.

I used to be advocate of link agg but after seeing the numbers for myself and the additional work in setting up the trunk group I am not so sure if it is worth the trouble.

How does "The receive load balancing is achieved by ARP negotiation" work? - Is it sending G-ARPS or what. And wont the "flow" be affected if chaning mac adress in the middle of the stream? something you dont have when LACP-ing?

/Christian

---

[Emilo]

Hello Chritian and thanks for reading and commenting on my post.

If you have Link Agg setup and are happy with the results continue to use it. I am not trying to convince anyone to switch from one bonding method to the other. However with the implementation of Saniq 9.0 and above uses balance-alb or 6

http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding#Bonding_Driver_Options

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation

When an ALB bond is configured, it creates an interface. This interface balances traffic through both nics. But how will this work with the iSCSI protocol? In RFC 3270 (http://www.ietf.org/rfc/rfc3720.txt) iSCSI uses command connection allegiance;

For any iSCSI request issued over a TCP connection, the corresponding response and/or other related PDU(s) MUST be sent over the same connection.  We call this “connection allegiance”. 

Hope this helps

You can also see this if you look at the logs just look histifconfig.log

[CRiiser]

Hmm,

reading the referred article I get the feeling that the loadbalancing is done per IP / ARP - thus multiple IPs spread over the nic:s in the bond.

Thus, if one IP needs more than 1GbE it wont get that if not using LACP bond. I.e. presenting one MAC all-the-time.

Right?

/Christian

[Emilo]

All bonds only use 1GB at a time or at the same time no bond will give 2GB.

[CRiiser]

Hmm. This im not so sure of. My impression is that when going LACP - you get an NEW MAC adress represented - from witch both send and receive is done. And this allows you to fully use BOTH nics bandwith (ie. 2GbE) in communicating. both receive wise and send wise...

As ALB will give you 2 GbE send and only 1GbE at max receive....

Would you say otherwise?

Kind regards,

Christian