Bonding Question

[Star Dust]

I'm having problem with a DL380 server running Redhat 4.5 that I'm trying to bond 4 broadcomm adapters together but it doesn't seem to be as fast as I'd hoped. I'm starting to think that I misunderstood the use of bonding. I was thinking that bonding would give me a massive network pipe. Is bonding more for failover rather then speed?

I've done a few tests between two servers each with four NIC cards bonded and the data transfer rate is 40mbs while using no bonding it's 50mbs.

Would someone mind explaining it to a simpleton? :)

[Ivan Ferreira]

Bonding provides round robin (mode=0) for link aggregation, but the switch must be configured to allow this. AFAIK, cisco calls to this EtherChannel.

[melvyn burnard]

also be aware that you can configure this for EITHER failover or all NIC's active.
check your configuration

[Matti_Kurkela]

Bonding can provide more bandwidth, which is not quite the same thing as increased speed.

It's the equivalent of adding more lanes to a highway without raising the speed limit: four lanes (NICs) can accommodate more trucks (packets) than one, but a single truck (packet) won't get from Point A to Point B any faster than before. The increased complexity may even make it a bit slower.

Of course, if the original single lane (NIC) was badly congested, adding more lanes can improve the situation for single trucks (packets) too.

If your test was designed specifically to measure speed, it may have failed to utilize the increased bandwidth: try running multiple copies of your test in parallel, then see the results.

MK

[Star Dust]

Wow... great explanation... That really helps...

I have created an EitherChannel on my switch, but I'm not sure if I set it up correctly. I took the four ports that I have my servers connected into and put them in Either-group 1 using the following command:

"Switch (config-if)#channel-group 1 mode on"

Does that look right or am I supposed to use mode auto or desirable?

Thanks again for all the help guys and gals... :)

[rick jones]

(ab)using mode-rr to increase the performance of a single connection has, well, "issues..." the trafic _will_ become reordered, and if it becomes sufficiently reordered, it triggers TCP's "fast" lost segment detection. This then supresses the size of the congestion window calculated by the sender, which leads to poor throughput. This is coupled with the vast increase in TCP ACKs from all the out-of-order traffic.

In my opinion the best way to increase the speed of a single connection is to upgrade to the next faster NIC type. Admittedly that isn't always possible, but it is better.

You _might_ get some relief by setting net.ipv4.tcp_reordering to something rather larger than the number of NICs in the bond but I consider that little more than a kludge.

[Star Dust]

Thanks for the comment! What is the next NIC type that I can use? 10gigE? Do your saying that I should go back down to using 2 NIC's instead of four because it's creating too much traffic for the server to sort through?

Thanks for the help!

[rick jones]

If you are using 1 gig then indeed the next step up would be 10 gig :)

If you do stick with mode-rr bonding, you could indeed try using just two links in the bond rather than four and see if things are better. Or you can try tweaking that sysctl. As you try things-out, keep looking at the netstat -t (iirc that is the syntax for TCP stats) stats.

It may also be necessary/desirable to enable larger socket buffers/windows to get things going faster.

Having said all that, it seems that without the bonding, you are still only running at half of link rate on the single connection test. I think it would be good to diagnose that further.

[Star Dust]

OK, I've narrowed down problem here. It looks like after I created a bond, RHEL will sometimes rearrange them on reboot for some reason. For example:

eth0 = IPADDRESS=10.10.10.20
eth1 = onboot=no
Bond0 = eth2, eth3, eth4, eth5

After I reboot the machine the the ifcfg files stay the same, but the hardware will move around! In other words eth0 may be eth4 on the next reboot which is why my bonding isn't working. I figured this out after disabling the embedded NIC adapters (eth0 and eth1) and now my bond works beautify!

Why is it doing this and is it possible to assign hardware to use a certain ifcfg file?

[rick jones]

is there a MAC address in each of the ifcfg files? if so then that _should_ be causing each interface to have the same name after each boot.

[Matti_Kurkela]

Note that in RHEL4 or above, the ifcfg files can have two directives with MAC addresses, with very different meanings:

DEVICE=eth0
...
HWADDR=

causes the device identified by the to be automatically renamed to eth0, if it isn't that by default.

DEVICE=eth1
...
MACADDR=
changes the MAC address of device eth1 to .

Mixing up these two is likely to cause great confusion.

If you're using HWADDR to rename an interface to e.g. eth0 and another interface is already named as eth0, it will get a temporary name, which will look very strange. So if you use this feature to re-arrange your network interfaces, specify the correct HWADDR directives for _all_ network interfaces of your system.

If you remove all HWADDR directives from the ifcfg files, the network configuration will again work like RHEL3 and classic RedHats.

MK

[Star Dust]

You guys rock... I added the HWADDR= into each file and now it seems stable as a rock... :) Thank you all very much for the help!

I'm now doing a little testing with it and wondered if I need to add "options bond0 miimon=100 mode=balance-rr" to my modprobe.conf file. Some documentation says to do it and some doesn't at all.

Also, is there anywhere else that I can do some tweaking?