Bonding and switch config problems

[kheller2]

I'm having issues getting bonding to correctly work with a pair of cisco 6500s where the vlan spans the two switches. The host is connected to each switch and when running with HP drivers for bonding and bcm5700 (DL585s) I get issues with the bond stating its up, then down, then up all day long -- which causes the HP health agents to reboot he box in the middle of the night. Using mode=1 and miimon=100. I also get duplicate packets back when I leave the subnet.

If I use mode=0 and straight RH3U5 drivers I don't see that issue. And technically, I'm told you can't do mode0 accross switches.

So, what am I missing or should I look for...

I have duplicate packets and bond interface bouncing all the time

Thanks..

[Steven E. Protter]

Shalom,

To be really helpful, I'd like to see the ifcfg-bond0 and two slave configuration files. Feel free to alter the ip address, but make them consistent.

I'd also like to see the /etc/modules.conf file and any relavent entries on the dmesg or /var/log/messages file.

Lastly, what are the following output.

ethtool eth0
ethtool eth1
ethtool bond0

I don't really expect anything meaningful from the last command, but the other commands can help me diagnose the problem.

As far as mode=0 goes, if it works and is stable, go for it. It may be the cause of the issue at hand.

SEP

[kheller2]

alias eth0 bcm5700
alias eth1 bcm5700
alias eth2 e1000
alias eth3 e1000
alias scsi_hostadapter lpfc
alias scsi_hostadapter1 cciss
alias usb-controller usb-ohci
below bonding e1000 bcm5700
below bonding1 e1000 bcm5700
probe all bond1 bond0 eth3 eth1 eth2 eth0 bonding
alias bond0 bonding
options bond0 miimon=100 mode=1 primary=eth0
alias bond1 bonding
options bond1 -o bonding1 miimon=120 mode=1 primary=eth1
options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
options scsi_mod max_scsi_luns=255 scsi_allow_ghost_devices=1
options lpfc lpfc_topology=0x02
alias ppfilter emcpsf
pre-install emcpsf modprobe sd_mod
insmod_opt=-N
alias power_path emcp
add above emcp emcpmp
add above emcpmp emcpmpc
add above emcpmpc emcpmpaa
add above emcpmpaa emcpmpap
add above emcpmpap emcphr
add above emcphr emcpioc
post-install emcpioc rmmod emcpioc
pre-remove emcpsf rmmod emcpmpaa emcpmpc emcpmp emcp
add options emcp managedclass="ess","hitachi","hpxp","hphsx"

[root@ashprdrptdb01 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
[root@ashprdrptdb01 network-scripts]# cat ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond1
SLAVE=yes
[root@ashprdrptdb01 network-scripts]# cat ifcfg-eth2
DEVICE=eth2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
[root@ashprdrptdb01 network-scripts]# cat ifcfg-eth3
DEVICE=eth3
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond1
SLAVE=yes

[root@ashprdrptdb01 network-scripts]# cat ifcfg-bond0
DEVICE=bond0
IPADDR=172.17.120.23
NETMASK=255.255.255.0
NETWORK=172.17.120.0
BROADCAST=172.21.6.255
ONBOOT=YES
BOOTPROTO=none
USERCTL=no
[root@ashprdrptdb01 network-scripts]# cat ifcfg-bond1
DEVICE=bond1
IPADDR=172.17.99.49
NETMASK=255.255.255.240
BROADCAST=172.17.99.63
ONBOOT=YES
BOOTPROTO=none
USERCTL=no

[kheller2]

Feb 13 12:12:19 ashprdrptdb01 kernel: bond1: link status definitely down for interface eth3, disabling it.
Feb 13 12:12:19 ashprdrptdb01 kernel: bond1: link status definitely up for interface eth3.
Feb 13 13:15:40 ashprdrptdb01 kernel: bond0: link status definitely down for interface eth2, disabling it.
Feb 13 13:15:41 ashprdrptdb01 kernel: bond0: link status definitely up for interface eth2.
Feb 13 13:59:56 ashprdrptdb01 kernel: bond1: link status definitely down for interface eth3, disabling it.
Feb 13 13:59:56 ashprdrptdb01 kernel: bond1: link status definitely up for interface eth3.


[root@ashprdrptdb01 network-scripts]# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
[root@ashprdrptdb01 network-scripts]# ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
[root@ashprdrptdb01 network-scripts]# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
[root@ashprdrptdb01 network-scripts]# ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
[root@ashprdrptdb01 network-scripts]# ethtool bond0
Settings for bond0:
No data available
[root@ashprdrptdb01 network-scripts]# ethtool bond1
Settings for bond1:
No data available

[kheller2]

[root@ashprdrptdb01 network-scripts]# rpm -qa |grep bond
bonding-1.0.4w-1
[root@ashprdrptdb01 network-scripts]# rpm -qa |grep bcm
bcm5700-8.2.14a-1
[root@ashprdrptdb01 network-scripts]# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 3 (Taroon Update 5)
[root@ashprdrptdb01 network-scripts]# uname -a
Linux ashprdrptdb01.blah.blah 2.4.21-32.ELhugemem #1 SMP Fri Apr 15 21:04:31 EDT 2005 i686 athlon i386 GNU/Linux

[Steven E. Protter]

Shalom,

Thanks for the information.

One things stands out for me.

eth0 is bcm5700
eth1 is bcm5700

eth0 and eth1 should be slaves to a single master bond0 or bond1

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
[root@ashprdrptdb01 network-scripts]# cat ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
MASTER=bond1


eth0 's master is bond0
eth1 's master is bond1

You have two e1000 cards in there, which are eth2 and eth3.

Perhaps they bond to bond1.

Bottom line is that bcm5700 can only be part of a bond pair with the same nic, ie another bcm5700.

Same goes for Intel.

I think this is the source of the entire problem. Either that or its too late for me to read.

SEP

[kheller2]

I'll check the bonding driver notes again, but I don't ever remember seeing a statement that you can't bond accross the chipsets when the bonding driver supports both. This is even more strange when you can run them with the tg3 driver.

[Steven E. Protter]

I think bonding across chipsets is a bad idea and is not supported by the OS.

For two cards to act as one reliably, they should be the same card and have the same firmware, etc.

Just for grins, give my concept a try and see if it helps. My experience says it will help.

:-)

I will be happily proven wrong.

SEP