bonding mode=1 switch port flapping when idle

[Tim Nelson]

Configured bonding in mode=1 ( active/backup ).
Last night while the server was idle my network admin noticed port flapping errors on the switch.

Once the traffic to the server picked up, the messages stopped.

Nothing special configured.

RHEL 5.3
alias bond0 bonding
options bond0 miimon=100 mode=1

[Steven E. Protter · Accepted solution]

Shalom,

First time I've seen this. I didn't choose mode=1 last time I bonded.

What kind of switch are we talking about? Another mode might alleviate the issue if the switch admin cares.

SEP

[Tim Nelson]

Hey SEP,

Cisco something or other.. ( will find out ).

Currently we are connected to a single switch and could do mode 6, but in a week or two these connections will be on separate switches and hence the need for active/backup mode 1.

The messages only occured last night while the connecitons were idle. I have no message log entries that the slaves were flip-flopping back and forth so I am guessing it had something to do with the miimon ( but that is suppose to only check for link up/down ).

I was looking at the fail_over_mac settings but the version support for that is not mainstream yet.

[Steven E. Protter]

I think the server OS is churning the switch as it checks that both paths are still available.

There is not a lot you can do other than perhaps configure it active/active, which would probably end the mode switch. If the switch supports it this would result in an 80% increase in overall band width. Sometimes with Cisco, switch configuration must be changed to support channel bonding active-active. Results also depend on type of NIC. I've had good results with Intel NIC, explosive results with broadcomm, though people say broadcomm is better now.

This is really interesting. Please keep us up to date.

SEP

[Tim Nelson]

Some supporing info: there are only 2 tx packers on eth1, this is good, i think.. Checking with our network group to see if the flapping when away.(if so I will post what I did).

snocvnaesb01@root:/root # ifconfig
bond0 Link encap:Ethernet HWaddr D8:D3:85:E0:87:04
inet addr:xxx.xxx.xxx.xxx
Bcast:xxx.xxx.xxx.0 Mask:255.255.255.0
inet6 addr: fe80::dad3:85ff:fee0:8704/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:1556400 errors:0 dropped:0 overruns:0 frame:0
TX packets:2797 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:156275046 (149.0 MiB) TX bytes:682678 (666.6 KiB)

eth0 Link encap:Ethernet HWaddr D8:D3:85:E0:87:04
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:779420 errors:0 dropped:0 overruns:0 frame:0
TX packets:2795 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:78273410 (74.6 MiB) TX bytes:682520 (666.5 KiB)
Interrupt:130 Memory:f8000000-f8012100

eth1 Link encap:Ethernet HWaddr D8:D3:85:E0:87:04
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:776980 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:78001636 (74.3 MiB) TX bytes:158 (158.0 b)
Interrupt:138 Memory:f6000000-f6012100

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1880 errors:0 dropped:0 overruns:0 frame:0
TX packets:1880 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4631101 (4.4 MiB) TX bytes:4631101 (4.4 MiB)

[Tim Nelson]

No more flapping reported...

I added the downdelay, updelay and primary= entries after reading some old threads about bonding defaulting to round-robin / not taking the options even though /proc/net/bonding/bond0 says it is correct.

alias bond0 bonding
options bonding mode=1 miimon=100 downdelay=200 updelay=200 max_bonds=1 primary=eth0

Also rebooted.

There just might be some bug in bonding but if I am the only one having the issue then maybe it was the way I enabled it.

e.g. edit modprobe.conf then "service network restart" instead of rebooting ? dunno