topic Re: Bonding Question in Operating System - Linux

Bonding Question

Star Dust — Fri, 21 Dec 2007 05:34:01 GMT

I'm having problem with a DL380 server running Redhat 4.5 that I'm trying to bond 4 broadcomm adapters together but it doesn't seem to be as fast as I'd hoped. I'm starting to think that I misunderstood the use of bonding. I was thinking that bonding would give me a massive network pipe. Is bonding more for failover rather then speed?

I've done a few tests between two servers each with four NIC cards bonded and the data transfer rate is 40mbs while using no bonding it's 50mbs.

Would someone mind explaining it to a simpleton? :)

Re: Bonding Question

Ivan Ferreira — Fri, 21 Dec 2007 13:10:11 GMT

Bonding provides round robin (mode=0) for link aggregation, but the switch must be configured to allow this. AFAIK, cisco calls to this EtherChannel.

Re: Bonding Question

melvyn burnard — Fri, 21 Dec 2007 13:48:26 GMT

also be aware that you can configure this for EITHER failover or all NIC's active.
check your configuration

Re: Bonding Question

Matti_Kurkela — Fri, 21 Dec 2007 14:34:45 GMT

Bonding can provide more bandwidth, which is not quite the same thing as increased speed.

It's the equivalent of adding more lanes to a highway without raising the speed limit: four lanes (NICs) can accommodate more trucks (packets) than one, but a single truck (packet) won't get from Point A to Point B any faster than before. The increased complexity may even make it a bit slower.

Of course, if the original single lane (NIC) was badly congested, adding more lanes can improve the situation for single trucks (packets) too.

If your test was designed specifically to measure speed, it may have failed to utilize the increased bandwidth: try running multiple copies of your test in parallel, then see the results.

MK

Re: Bonding Question

Star Dust — Fri, 21 Dec 2007 15:28:15 GMT

Wow... great explanation... That really helps...

I have created an EitherChannel on my switch, but I'm not sure if I set it up correctly. I took the four ports that I have my servers connected into and put them in Either-group 1 using the following command:

"Switch (config-if)#channel-group 1 mode on"

Does that look right or am I supposed to use mode auto or desirable?

Thanks again for all the help guys and gals... :)

Re: Bonding Question

rick jones — Mon, 31 Dec 2007 19:02:30 GMT

(ab)using mode-rr to increase the performance of a single connection has, well, "issues..." the trafic _will_ become reordered, and if it becomes sufficiently reordered, it triggers TCP's "fast" lost segment detection. This then supresses the size of the congestion window calculated by the sender, which leads to poor throughput. This is coupled with the vast increase in TCP ACKs from all the out-of-order traffic.

In my opinion the best way to increase the speed of a single connection is to upgrade to the next faster NIC type. Admittedly that isn't always possible, but it is better.

You _might_ get some relief by setting net.ipv4.tcp_reordering to something rather larger than the number of NICs in the bond but I consider that little more than a kludge.

Re: Bonding Question

Star Dust — Wed, 02 Jan 2008 20:08:12 GMT

Thanks for the comment! What is the next NIC type that I can use? 10gigE? Do your saying that I should go back down to using 2 NIC's instead of four because it's creating too much traffic for the server to sort through?

Thanks for the help!

Re: Bonding Question

rick jones — Wed, 02 Jan 2008 20:53:00 GMT

If you are using 1 gig then indeed the next step up would be 10 gig :)

If you do stick with mode-rr bonding, you could indeed try using just two links in the bond rather than four and see if things are better. Or you can try tweaking that sysctl. As you try things-out, keep looking at the netstat -t (iirc that is the syntax for TCP stats) stats.

It may also be necessary/desirable to enable larger socket buffers/windows to get things going faster.

Having said all that, it seems that without the bonding, you are still only running at half of link rate on the single connection test. I think it would be good to diagnose that further.

Re: Bonding Question

Star Dust — Thu, 03 Jan 2008 23:35:39 GMT

OK, I've narrowed down problem here. It looks like after I created a bond, RHEL will sometimes rearrange them on reboot for some reason. For example:

eth0 = IPADDRESS=10.10.10.20
eth1 = onboot=no
Bond0 = eth2, eth3, eth4, eth5

After I reboot the machine the the ifcfg files stay the same, but the hardware will move around! In other words eth0 may be eth4 on the next reboot which is why my bonding isn't working. I figured this out after disabling the embedded NIC adapters (eth0 and eth1) and now my bond works beautify!

Why is it doing this and is it possible to assign hardware to use a certain ifcfg file?

Re: Bonding Question

rick jones — Fri, 04 Jan 2008 00:51:54 GMT

is there a MAC address in each of the ifcfg files? if so then that _should_ be causing each interface to have the same name after each boot.

Re: Bonding Question

Matti_Kurkela — Fri, 04 Jan 2008 12:45:39 GMT

Note that in RHEL4 or above, the ifcfg files can have two directives with MAC addresses, with very different meanings:

DEVICE=eth0
...
HWADDR=

causes the device identified by the to be automatically renamed to eth0, if it isn't that by default.

DEVICE=eth1
...
MACADDR=
changes the MAC address of device eth1 to .

Mixing up these two is likely to cause great confusion.

If you're using HWADDR to rename an interface to e.g. eth0 and another interface is already named as eth0, it will get a temporary name, which will look very strange. So if you use this feature to re-arrange your network interfaces, specify the correct HWADDR directives for _all_ network interfaces of your system.

If you remove all HWADDR directives from the ifcfg files, the network configuration will again work like RHEL3 and classic RedHats.

MK

Re: Bonding Question

Star Dust — Sat, 05 Jan 2008 16:29:26 GMT

You guys rock... I added the HWADDR= into each file and now it seems stable as a rock... :) Thank you all very much for the help!

I'm now doing a little testing with it and wondered if I need to add "options bond0 miimon=100 mode=balance-rr" to my modprobe.conf file. Some documentation says to do it and some doesn't at all.

Also, is there anywhere else that I can do some tweaking?