RHEL 4.5 Bonding troubles

[Star Dust]

I'm posting this again...

I have two servers, each with 6 NIC cards in them. The first two NIC's are for metadata and the last 4 (eth2 through eth5) I want bonded. All cards use the bnx2 driver and I've tried the HP version and the standard Redhat version with no difference.

I think I have the server side and switch side configured correct, but I'd like someone opinion because it's not working right. When I try to ping or move files from one machine to another there is very high latency or they don't talk at all. I'm not sure what I'm doing wrong. Here are the configurations that I'm using as well as some output from the /var/log/messages file. Thanks for the help!

############# Switch Settings #######
interface Port-channel1
description bonding1
switchport access vlan 1
switchport mode access
no snmp trap link-status

interface Port-channel2
description bonding2
switchport access vlan 1
switchport mode access
no snmp trap link-status


interface range GigabitEthernet 0/1-4
description bonding1
switchport access vlan 1
switchport mode access
no snmp trap link-status
channel-group 1 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable

interface range GigabitEthernet 0/5-8
description bonding2
switchport access vlan 1
switchport mode access
no snmp trap link-status
channel-group 2 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable

############# Switch Settings #######

I'm going to have to go back into work and get my ifcfg-bon0 settings since my post was deleted last night, but it's pretty standard and when when I only had two NIC's bonded (which worked) it what I used. I also do have the "alias bond0 bonding" in my /etc/modprobe.conf file along with all the NIC cards.

###################### /var/log/messages

san-03 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
san-03 kernel: ip_tables: (C) 2000-2002 Netfilter core team
san-03 kernel: bnx2: eth2: using MSI
san-03 kernel: ADDRCONF(NETDEV_UP): eth2: link is not ready
san-03 kernel: bonding: Warning: failed to get speed/duplex from eth2, speed forced to 100Mbps, duplex f orced to Full.
san-03 kernel: bonding: bond0: enslaving eth2 as an active interface with an up link.
san-03 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
san-03 kernel: bnx2: eth3: using MSI
san-03 kernel: ADDRCONF(NETDEV_UP): eth3: link is not ready
san-03 kernel: bonding: Warning: failed to get speed/duplex from eth3, speed forced to 100Mbps, duplex f orced to Full.
san-03 kernel: bonding: bond0: enslaving eth3 as an active interface with an up link.
san-03 kernel: bnx2: eth4: using MSI
san-03 kernel: ADDRCONF(NETDEV_UP): eth4: link is not ready
san-03 kernel: bonding: Warning: failed to get speed/duplex from eth4, speed forced to 100Mbps, duplex f orced to Full.
san-03 kernel: bonding: bond0: enslaving eth4 as an active interface with an up link.
san-03 kernel: bnx2: eth5: using MSI
san-03 kernel: ADDRCONF(NETDEV_UP): eth5: link is not ready
san-03 kernel: bonding: Warning: failed to get speed/duplex from eth5, speed forced to 100Mbps, duplex f orced to Full.
san-03 kernel: bonding: bond0: enslaving eth5 as an active interface with an up link.
san-03 kernel: bnx2: eth2 NIC Copper Link is Up, 1000 Mbps full duplex
san-03 kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
san-03 kernel: bnx2: eth3 NIC Copper Link is Up, 1000 Mbps full duplex
san-03 kernel: bnx2: eth4 NIC Copper Link is Up, 1000 Mbps full duplex
san-03 kernel: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready
san-03 kernel: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

###################### /var/log/messages

[skt_skt]

are the NIC cards and the switches support 1000MBPS?(if yes you make sure that duplex is full and auto negotitaion is on at both sides.



If you are unsure what is configured at the switch side contact your netowrk team to confirm that.. I could not follow anyhting about the speed and duplex from what you posted under "switch settings"

from the messages file you posted it looks like auto negotiation is off and speed to got set to 100MBPS and dulex to full ... later it is reporting 1000 Mbps full duplex

could you post "ethtool ethx" for all where x is the respective cards

[Steven E. Protter]

Shalom,

Bonding was really designed and tested for pairing two cards.

Though obviously possible to bond more than two, the results you receive may be due to one of the following factors:

problems in /etc/modprobe.conf # typo?
You are pushing the software beyond where the developers tested it.
Network switch/port settings prevent proper bonding. On cisco switches some assembly is required for bonding to work.

SEP

[Star Dust]

Thank you both for the responses. I've directly cut and copy my config files from my linux machines and from my switch as well as the ethtool outputs. I did notice that for some reason my eth5 card cannot be read. What would cause something like that to happen? It's the second port from a duel port NIC card.


####################### Cisco settings

interface Vlan1
ip address 192.168.1.215 255.255.255.0
!
interface Vlan2
ip address 10.10.10.1 255.255.255.0


interface Port-channel1
description BOND1
switchport mode access
no snmp trap link-status
!
interface Port-channel2
description BOND2
switchport mode access
no snmp trap link-status
!
interface Port-channel3
description BOND3
switchport mode access
no snmp trap link-status
!
interface GigabitEthernet0/1
description SAN1-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 1 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/2
description SAN1-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 1 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/3
description SAN1-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 1 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/4
description SAN1-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 1 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/5
description SAN2-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 2 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/6
description SAN2-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 2 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/7
description SAN2-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 2 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/8
description SAN2-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 2 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/9
description SAN3-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 3 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/10
description SAN3-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 3 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/11
description SAN3-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 3 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable
!
interface GigabitEthernet0/12
description SAN3-BOND
switchport mode access
speed 1000
duplex full
no snmp trap link-status
channel-group 3 mode on
spanning-tree portfast disable
spanning-tree bpduguard disable

interface GigabitEthernet0/45
description MetaData San1
switchport access vlan 2
switchport mode access
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
switchport port-security aging type inactivity
macro description cisco-desktop
spanning-tree portfast
spanning-tree bpduguard enable
!
interface GigabitEthernet0/46
description MetaData San2
switchport access vlan 2
switchport mode access
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
switchport port-security aging type inactivity
macro description cisco-desktop
spanning-tree portfast
spanning-tree bpduguard enable
!
interface GigabitEthernet0/47
description MetaData San3
switchport access vlan 2
switchport mode access
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
switchport port-security aging type inactivity
macro description cisco-desktop
spanning-tree portfast
spanning-tree bpduguard enable
!
interface GigabitEthernet0/48
description MetaData San4
switchport access vlan 2
switchport mode access
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
switchport port-security aging type inactivity
macro description cisco-desktop
spanning-tree portfast
spanning-tree bpduguard enable


####################### Cisco settings


####################### /etc/modprobe.conf

alias eth0 bnx2
alias eth1 bnx2
alias scsi_hostadapter qla2xxx
alias scsi_hostadapter1 cciss
alias scsi_hostadapter2 ata_piix
alias scsi_hostadapter3 qla2400
alias usb-controller uhci-hcd
alias usb-controller1 ehci-hcd
alias eth3 bnx2
alias eth2 bnx2
alias eth4 bnx2
alias eth5 bnx2
alias bond0 bonding

####################### /etc/modprobe.conf

####################### ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.1.212
NETMASK=255.255.255.0
NETWORK=192.168.1.0
BROADCAST=192.168.1.255
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
IPV6INIT=no
PEERDNS=yes
TYPE=Ethernet

####################### ifcfg-bond0


####################### ifcfg-ethx

DEVICE=ethx
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
TYPE=Ethernet
IPV6INIT=no


####################### ifcfg-ethx


####################### ethtool

Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
Settings for eth4:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes
Settings for eth5:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: Unknown! (65535)
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: no

####################### ethtool

[skt_skt]

for eth5 there is no link detected. Could be a cable not/loose connected or a defective cable itself.. verify from both server and switch side.

Settings for eth5:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: Unknown! (65535)
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: no