Problems with bonding on bl20p

[Erin Simonson]

We have several bl20p's that we are trying to get bonding to work on. It seems to be setup properly and working but the speed is slower than not having bonding setup. We also have the following message in dmesg.
bonding: Warning: failed to get speed/duplex from eth2, speed forced to 100Mbps, duplex forced to Full.
bonding: bond0: enslaving eth2 as an active interface with an up link.
Here is the output from ifconfig if it helps.
bond0 Link encap:Ethernet HWaddr 00:11:85:50:7E:9C
inet addr:192.168.100.20 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:31413841 errors:0 dropped:0 overruns:0 frame:0
TX packets:42091927 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3909315614 (3.6 GiB) TX bytes:381088644 (363.4 MiB)

eth0 Link encap:Ethernet HWaddr 00:11:85:50:7E:9C
inet6 addr: fe80::211:85ff:fe50:7e9c/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1544856 errors:0 dropped:0 overruns:0 frame:0
TX packets:21045964 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:250083146 (238.4 MiB) TX bytes:190966131 (182.1 MiB)
Interrupt:185 Memory:f7df0000-f7e00000

eth1 Link encap:Ethernet HWaddr 00:11:0A:EA:B9:1C
inet addr:172.31.1.5 Bcast:172.31.1.255 Mask:255.255.255.0
inet6 addr: fe80::211:aff:feea:b91c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:29822387 errors:0 dropped:0 overruns:0 frame:0
TX packets:17218753 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:804623236 (767.3 MiB) TX bytes:1666842641 (1.5 GiB)
Interrupt:193 Memory:f7de0000-f7df0000

eth2 Link encap:Ethernet HWaddr 00:11:85:50:7E:9C
inet6 addr: fe80::211:85ff:fe50:7e9c/64 Scope:Link
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:29868985 errors:0 dropped:0 overruns:0 frame:0
TX packets:21045963 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3659232468 (3.4 GiB) TX bytes:190122513 (181.3 MiB)
Interrupt:217 Memory:f7ff0000-f8000000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:12232222 errors:0 dropped:0 overruns:0 frame:0
TX packets:12232222 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4233391384 (3.9 GiB) TX bytes:4233391384 (3.9 GiB)

Thanks for any help you can give me.

[Steven E. Protter]

Shalom Erin,

mii-tool commonly gives incorrect speed readings on bonded cards.

ethtool bond0

Should show a more accurate speed assessment.

Please give details on which configuration files you modified to do the bonding.

modules.conf ??

ifcfg-eth0 ... ??

SEP

[Ivan Ferreira]

What are the parameters (mode=) that you use when load the bonding module?

What are the performance metrics that you use?

The speed for the network adapters has been set correctly 100 Full duplex, you should ensure that in the switch, the speed is the same (It should be as you don't have packet errors and collisions)

It seems that eth0 and eth2 are in the bond0 device. What is the output of

ethtool eth0
ethtool eth2

[Erin Simonson]

Here is the output from ethtool
[root@wi00-00 tmp]# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes

The files I modified were ifcfg-eth0, ifcfg-eth2, and modules.conf.

[Ivan Ferreira]

Please post the the configuration of all files, and the ethtool output of both ethernet adapters.

[Erin Simonson]

[root@wi00-00 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
You have new mail in /var/spool/mail/root
[root@wi00-00 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[root@wi00-00 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.100.20
NETMASK=255.255.255.0
NETWORK=192.168.100.0
BROADCAST=192.168.100.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
[root@wi00-00 ~]# cat /etc/modprobe.conf
alias eth0 tg3
alias eth1 tg3
alias eth2 tg3
alias bond0 bonding
options bond0 miimon=100
alias scsi_hostadapter cciss
alias scsi_hostadapter1 qla2300
alias usb-controller ohci-hcd
[root@wi00-00 ~]# ethtool eth0
Settings for eth0:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes
[root@wi00-00 ~]# ethtool eth2
Settings for eth2:
Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes
[root@wi00-00 ~]#

[Ivan Ferreira]

Edit the modules.conf and set the mode for the bond device:

options bond0 miimon=100 mode=1


Where mode:

0 = round-robin
1 = failover


Also, you can set the speed for the network adapters manually editing the ifcfg-ethX file, add the following:

ETHTOOL_OPTS="speed 1000 duplex full autoneg off"

If the ethernet devices are connected to 1000/Mb switches, or:

ETHTOOL_OPTS="speed 100 duplex full autoneg off"

If are connected to 100 Mb switches.

[Steven E. Protter]

OK: Good info.

If the last post before this fails to solve the issue then see if the network hardware is really GB. If its configurable, make sure its switch port settings are to full duplex and autonegotiate.

SEP

[Erin Simonson]

I set the options bond0 miimon=100 mode=1 in modprobe.conf and added these ETHTOOL_OPTS="speed 1000 duplex full autoneg off" to both ifcfg-ethX files and still no luck. The switch is set to full duplex and autonegotiate.

[Ivan Ferreira]

Have you restarted the server after the changes? Is needed to reload the modules (easy way). The changes where seen by ethtool, run ethtool eth0 and eth2 again after the restart. Also, if you disabled the autonegotiation on the adapter, you should disable it on the switch, and configure manually to the same speed. Also, verify if the switch port has errors and collisions in the statistics.

How do you measure the performance?

[Erin Simonson]

I have restarted the machine and ethtool shows
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full

I set autonegotiate to on in the ETHTOOLS_OPTS since i dont have physical access to the switches config from here.

I am measureing performance doing a nc from one machine to a machine with bonding enabled and to a machine without it. I seem to get about the same speed with or without bonding.