topic Re: Network Bonding in Operating System - Linux

Network Bonding

faisal chaudry — Sat, 10 Jul 2010 06:55:49 GMT

I have two DL380 G6 Server. Operating System is RHEL 4.7. Both servers have 2 quad port NICs installed in them. I have configured network bonding on these 2 servers.

Bonding is working just fine on Server1 but its not working on Server2.

All configurations have been copied from Server 1 to Server2.

Can any body tell me what could possibly wrong ?

Re: Network Bonding

Ivan Ferreira — Sat, 10 Jul 2010 13:00:59 GMT

What is "not working". Can you describe your tests and if possible, post the output of your configuration, ifconfig, ethtool, cat /proc/net/bonding/bondX... etc.

Re: Network Bonding

Alexander Chuzhoy — Sun, 11 Jul 2010 09:45:53 GMT

Hmm,
did you also copy the HWADDR or IPADDR fields in the configuration files?

Those fields must be unique.

Re: Network Bonding

faisal chaudry — Mon, 12 Jul 2010 04:36:58 GMT

Thanks for your response,

Actually when I assign IP Address directly to Ethernet interfaces, Every thing works fine. I can ping Servers, Switches & other network devices but when I configure bonding on servers, 1st Server is working fine, 2nd server is unable to ping 1st server & other network devices. 2nd server cannot even ping gateway.

I have copied all configuration script from 1st server on which everything is working fine.

[root@otasrv2 network-scripts]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth5
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:da:f7:9c
Slave Interface: eth9
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:da:f7:dc
---------------------------------------------
[root@otasrv2 network-scripts]# ethtool eth5
Settings for eth5:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes

Bond1 is configured with eth5 and eth9

When itâ s trying to get up in /var/log/messages we have:

Jul 9 18:04:19 otasrv2 kernel: bonding: Warning: failed to get speed/duplex from eth5, speed forced to 100Mbps, duplex forced to Full.

Jul 9 18:04:19 otasrv2 kernel: bonding: bond1: enslaving eth5 as a backup interface with an up link.

Jul 9 18:04:20 otasrv2 kernel: bonding: Warning: failed to get speed/duplex from eth9, speed forced to 100Mbps, duplex forced to Full.

Jul 9 18:04:20 otasrv2 kernel: bonding: bond1: enslaving eth9 as a backup interface with an up link.

When I try make eth5 active in bonding by following command:

[root@otasrv2 network-scripts]# ifenslave -c bond1 eth5

I got following error:
Master 'bond1', Slave 'eth5': Error: Change active failed

Re: Network Bonding

Alexander Chuzhoy — Mon, 12 Jul 2010 10:51:00 GMT

To permanently set the speed/duplex option -Append a line like the following in the NIC configuration file(s).

ETHTOOL_OPTS="speed 100 duplex full autoneg off"

BTW, It would help if you could attach the conf files for the bond/nics.

Re: Network Bonding

faisal chaudry — Mon, 12 Jul 2010 11:18:34 GMT

[root@otasrv2 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=static
IPADDR=172.16.33.57
NETMASK=255.255.255.0
GATEWAY=172.16.33.253

[root@otasrv2 network-scripts]# cat ifcfg-eth7
DEVICE=eth7
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

[root@otasrv2 network-scripts]# cat ifcfg-eth11
DEVICE=eth11
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

[root@otasrv2 network-scripts]# cat ifcfg-bond1
DEVICE=bond1
ONBOOT=yes
BOOTPROTO=none
IPADDR=172.16.171.221
NETMASK=255.255.255.0

[root@otasrv2 network-scripts]# cat ifcfg-eth5
DEVICE=eth5
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no
BOOTPROTO=none

[root@otasrv2 network-scripts]# cat ifcfg-eth9
DEVICE=eth9
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no
BOOTPROTO=none

[root@otasrv2 network-scripts]# cat /etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2
alias eth3 bnx2
alias eth4 e1000
alias eth5 e1000
alias eth6 e1000
alias eth7 e1000
alias eth8 e1000
alias eth9 e1000
alias eth10 e1000
alias eth11 e1000
alias scsi_hostadapter qla2xxx
alias scsi_hostadapter1 cciss
alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter3 qla2400
alias usb-controller uhci-hcd
alias usb-controller1 ehci-hcd
alias bond0 bonding
alias bond1 bonding
options bonding max_bonds=2 mode=1 miimon=100 primary=eth9
install vxportal /sbin/modprobe --ignore-install vxportal; /sbin/start_udev
alias char-major-10-32 vxportal

[root@otasrv2 network-scripts]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

[root@otasrv2 network-scripts]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth5
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:da:f7:9c

Slave Interface: eth9
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:da:f7:dc

[root@otasrv2 network-scripts]#tail -f /var/logs/messages
Jul 10 14:45:22 otasrv2 kernel: bonding: Warning: the permanent HWaddr of eth5 - 00:26:55:DA:F7:9C - is still in use by bond1. Set the HWaddr of eth5 to a different address to avoid conflicts.
Jul 10 14:45:22 otasrv2 kernel: bonding: bond1: releasing backup interface eth5
Jul 10 14:45:22 otasrv2 kernel: bonding: bond1: releasing backup interface eth9
Jul 10 14:45:23 otasrv2 network: Shutting down interface bond1: succeeded
Jul 10 14:45:23 otasrv2 network: Shutting down interface eth6: succeeded
Jul 10 14:45:23 otasrv2 network: Shutting down interface eth7: succeeded
Jul 10 14:45:23 otasrv2 network: Shutting down interface eth10: succeeded
Jul 10 14:45:23 otasrv2 network: Shutting down loopback interface: succeeded
Jul 10 14:45:23 otasrv2 sysctl: net.ipv4.ip_forward = 0
Jul 10 14:45:23 otasrv2 sysctl: net.ipv4.ip_local_port_range = 32768 61000
Jul 10 14:45:23 otasrv2 sysctl: net.ipv4.tcp_keepalive_time = 300
Jul 10 14:45:23 otasrv2 sysctl: net.ipv4.conf.default.rp_filter = 1
Jul 10 14:45:23 otasrv2 sysctl: net.ipv4.conf.default.accept_source_route = 0
Jul 10 14:45:23 otasrv2 sysctl: kernel.sysrq = 0
Jul 10 14:45:23 otasrv2 sysctl: fs.file-max = 327679
Jul 10 14:45:23 otasrv2 sysctl: kernel.core_uses_pid = 1
Jul 10 14:45:23 otasrv2 network: Setting network parameters: succeeded
Jul 10 14:45:23 otasrv2 network: Bringing up loopback interface: succeeded
Jul 10 14:45:24 otasrv2 kernel: ADDRCONF(NETDEV_UP): bond1: link is not ready
Jul 10 14:45:24 otasrv2 kernel: LLT INFO V-14-1-10205 link 1 (eth6) node 0 in trouble
Jul 10 14:45:25 otasrv2 kernel: LLT INFO V-14-1-10205 link 0 (eth10) node 0 in trouble
Jul 10 14:45:29 otasrv2 ifup: Enslaving eth5 to bond1
Jul 10 14:45:29 otasrv2 kernel: bonding: Warning: failed to get speed/duplex from eth5, speed forced to 100Mbps, duplex forced to Full.
Jul 10 14:45:29 otasrv2 kernel: bonding: bond1: enslaving eth5 as a backup interface with an up link.
Jul 10 14:45:29 otasrv2 kernel: ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
Jul 10 14:45:29 otasrv2 ifup: Enslaving eth9 to bond1
Jul 10 14:45:29 otasrv2 kernel: bonding: Warning: failed to get speed/duplex from eth9, speed forced to 100Mbps, duplex forced to Full.
Jul 10 14:45:29 otasrv2 kernel: bonding: bond1: enslaving eth9 as a backup interface with an up link.
Jul 10 14:45:29 otasrv2 network: Bringing up interface bond1: succeeded
Jul 10 14:45:29 otasrv2 kernel: ADDRCONF(NETDEV_UP): eth6: link is not ready
Jul 10 14:45:31 otasrv2 kernel: e1000: eth5: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 10 14:45:31 otasrv2 kernel: LLT INFO V-14-1-10032 link 1 (eth6) node 0 inactive 8 sec (520900)
Jul 10 14:45:31 otasrv2 kernel: e1000: eth9: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 10 14:45:31 otasrv2 kernel: e1000: eth6: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 10 14:45:31 otasrv2 kernel: ADDRCONF(NETDEV_CHANGE): eth6: link becomes ready
Jul 10 14:45:32 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 8 sec (979821)
Jul 10 14:45:32 otasrv2 kernel: LLT INFO V-14-1-10032 link 1 (eth6) node 0 inactive 9 sec (520900)
Jul 10 14:45:33 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 9 sec (979821)
Jul 10 14:45:33 otasrv2 kernel: LLT INFO V-14-1-10024 link 1 (eth6) node 0 active
Jul 10 14:45:34 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 10 sec (979821)
Jul 10 14:45:34 otasrv2 network: Bringing up interface eth6: succeeded
Jul 10 14:45:34 otasrv2 kernel: ADDRCONF(NETDEV_UP): eth7: link is not ready
Jul 10 14:45:35 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 11 sec (979821)
Jul 10 14:45:36 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 12 sec (979821)
Jul 10 14:45:36 otasrv2 kernel: e1000: eth7: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 10 14:45:36 otasrv2 kernel: ADDRCONF(NETDEV_CHANGE): eth7: link becomes ready
Jul 10 14:45:37 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 13 sec (979821)
Jul 10 14:45:37 otasrv2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth10) node 0. 4 more to go.
Jul 10 14:45:37 otasrv2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth10) node 0. 3 more to go.
Jul 10 14:45:38 otasrv2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth10) node 0. 2 more to go.
Jul 10 14:45:38 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 14 sec (979821)
Jul 10 14:45:38 otasrv2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth10) node 0. 1 more to go.
Jul 10 14:45:38 otasrv2 kernel: LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth10) node 0. 0 more to go.
Jul 10 14:45:39 otasrv2 kernel: LLT INFO V-14-1-10032 link 0 (eth10) node 0 inactive 15 sec (979821)
Jul 10 14:45:39 otasrv2 kernel: LLT INFO V-14-1-10509 link 0 (eth10) node 0 expired
Jul 10 14:45:39 otasrv2 network: Bringing up interface eth7: succeeded
Jul 10 14:45:39 otasrv2 kernel: ADDRCONF(NETDEV_UP): eth10: link is not ready
Jul 10 14:45:41 otasrv2 kernel: e1000: eth10: e1000_watchdog_task: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jul 10 14:45:41 otasrv2 kernel: ADDRCONF(NETDEV_CHANGE): eth10: link becomes ready
Jul 10 14:45:43 otasrv2 kernel: GAB INFO V-15-1-20036 Port h gen deeb25 membership 01
Jul 10 14:45:43 otasrv2 kernel: GAB INFO V-15-1-20037 Port h gen deeb25 jeopardy ;1
Jul 10 14:45:43 otasrv2 kernel: GAB INFO V-15-1-20036 Port a gen deeb28 membership 01
Jul 10 14:45:43 otasrv2 kernel: GAB INFO V-15-1-20037 Port a gen deeb28 jeopardy ;1
Jul 10 14:45:43 otasrv2 Had[29005]: VCS INFO V-16-1-10077 Received new cluster membership
Jul 10 14:45:43 otasrv2 Had[29005]: VCS ERROR V-16-1-10111 System otasrv2 (Node '1') is in Regular and Jeopardy Memberships - Membership: 0x3, Jeopardy: 0x2
Jul 10 14:45:44 otasrv2 network: Bringing up interface eth10: succeeded
Jul 10 14:45:46 otasrv2 kernel: LLT INFO V-14-1-10024 link 0 (eth10) node 0 active
Jul 10 14:45:51 otasrv2 kernel: GAB INFO V-15-1-20036 Port a gen deeb28 membership 01
Jul 10 14:45:51 otasrv2 kernel: GAB INFO V-15-1-20036 Port h gen deeb25 membership 01
Jul 10 14:45:51 otasrv2 Had[29005]: VCS INFO V-16-1-10077 Received new cluster membership
Jul 10 14:45:51 otasrv2 Had[29005]: VCS NOTICE V-16-1-10086 System otasrv2 (Node '1') is in Regular Membership - Membership: 0x3

Re: Network Bonding

Alexander Chuzhoy — Mon, 12 Jul 2010 11:30:37 GMT

Hmm, note this message:
Jul 10 14:45:22 otasrv2 kernel: bonding: Warning: the permanent HWaddr of eth5 - 00:26:55:DA:F7:9C - is still in use by bond1. Set the HWaddr of eth5 to a different address to avoid conflicts.

Try to create a fake MAC address for eth5 or bond1.

Re: Network Bonding

faisal chaudry — Mon, 12 Jul 2010 12:15:42 GMT

ok so if I enter fake HW in the configuration file of bond1 then it can resolve the issue ??

Re: Network Bonding

Alexander Chuzhoy — Mon, 12 Jul 2010 12:22:02 GMT

Sure hope so. :)

Re: Network Bonding

faisal chaudry — Wed, 14 Jul 2010 05:15:10 GMT

Hi Guys,

Thanks to all of you for support,

I just removed all configuration from srv2 then copied bonding configuration from srv1(which is working fine), then restarted network service but that did'nt resolve the issue, after that I just rebooted the srv2 & issue is resolved now.

Re: Network Bonding

faisal chaudry — Wed, 14 Jul 2010 05:17:04 GMT

Problem solved after reconfiguring bonding & reboot.