BL460c G7 - RHEL5 - Bonding of Emulex and Broadcom NICs

[Rob Leadbeater]

Hi,

Looking at the specs of the BL460c G7 I see that HP have moved to using Emulex adapters for the onboard NICs.

Has anyone had any experience of bonding the onboard NICs with a Broadcom mezzanine card - the NC382m, on Red Hat Enterprise Linux 5 ?

I've previously had issues bonding Broadcom and Intel cards, so just trying to preempt any potential issues.

Regards,

Rob

[edoardo75]

Hi, I've got and BL460c G7 with NC382m mezzanine.

I have a lot of issue on RHEL5 (update 6 and update 7 BETA) and Oracle EL 5.6. Mezzanine card driver of NC382m sometime is loaded before be2net driver (embedded card). Sometime eth0 is a Broadcom, sometime is a Emulex, furthermore bonding on NC382m do not work:

/etc/modprobe.conf

alias scsi_hostadapter shpchp
alias eth0 bnx2
alias eth1 bnx2
alias eth2 be2net
alias eth3 be2net
alias scsi_hostadapter1 cciss
alias scsi_hostadapter2 qla2xxx
alias scsi_hostadapter3 usb-storage
alias net-pf-10 off
alias ipv6 off
options ipv6 disable=1
alias bond0 bonding
options bond0 mode=1 miimon=100 use_carrier=0
options bonding max_bonds=2
alias bond1 bonding
options bond1 mode=1 miimon=100 use_carrier=0
options bonding max_bonds=2

ip link command output

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond1 qlen 1000
    link/ether 78:e3:b5:0e:da:94 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond1 qlen 1000
    link/ether 78:e3:b5:0e:da:94 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
    link/ether 78:e3:b5:09:de:78 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 qlen 1000
    link/ether 78:e3:b5:09:de:78 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 78:e3:b5:09:de:78 brd ff:ff:ff:ff:ff:ff
7: bond1: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
    link/ether 78:e3:b5:0e:da:94 brd ff:ff:ff:ff:ff:ff

[qvlatrt]

Hi to all,

I have similar bonding problem, but only with Emulex NIC, Broadcom is working correct:

Bonding miimon=100 for Emulex ethernet ports is not working. If I unplugg cable, Link will staid up (MII Status: up)and traffic will be losted. more /etc/sysconfig/hwconf class: NETWORK bus: PCI detached: 0 device: eth0 driver: be2net desc: "Emulex Corporation OneConnect 10Gb NIC (be3)" network.hwaddr: 9c:8e:99:1f:a1:a8 vendorId: 19a2 deviceId: 0710 subVendorId: 103c subDeviceId: 3315 pciType: 1 pcidom: 0 pcibus: 2 pcidev: 0 pcifn: 0 - class: NETWORK bus: PCI detached: 0 device: eth1 driver: be2net desc: "Emulex Corporation OneConnect 10Gb NIC (be3)" network.hwaddr: 9c:8e:99:1f:a1:ac vendorId: 19a2 deviceId: 0710 subVendorId: 103c subDeviceId: 3315 pciType: 1 pcidom: 0 pcibus: 2 pcidev: 0 pcifn: 1 - On another hand, if I configured bonding with arp_interval and arp_ip_target system recognized when cable is unplugged: cat /proc/net/bonding/bond0 shows that ,MII Status: down ifconfig -a show that interface is RUNNING - also BUG But traffic will not be losted and stand by ethenret interface will handle it. After I unplug cable back I will need to wait for more then 5 minutes to link become up - BUG Version-Release number of selected component (if applicable): modinfo be2net filename: /lib/modules/2.6.18-274.el5/kernel/drivers/net/benet/be2net.ko license: GPL author: ServerEngines Corporation description: ServerEngines BladeEngine 10Gbps NIC Driver 4.0.100r version: 4.0.100r srcversion: E7E2B568938BBCF1B40C2A4 alias: pci:v000010DFd0000E220sv*sd*bc*sc*i* alias: pci:v000019A2d00000710sv*sd*bc*sc*i* alias: pci:v000019A2d00000700sv*sd*bc*sc*i* alias: pci:v000019A2d00000221sv*sd*bc*sc*i* alias: pci:v000019A2d00000211sv*sd*bc*sc*i* depends: 8021q vermagic: 2.6.18-274.el5 SMP mod_unload gcc-4.1 parm: rx_frag_size:Size of a fragment that holds rcvd data. (ushort) parm: num_vfs:Number of PCI VFs to initialize (uint) parm: lro:Obsolete, only for backward compatibility. Don't use. (uint) parm: multi_rxq:Multi Rx Queue support. Enabled by default (uint) module_sig: 883f3504e177a195e46ee793d43754112da8609ca856df1cf8a119e1ee5f2baad81bb7436c698509b598dd774cc3d5a7cd1fb9049acf538768afe95 Configuration: cat /etc/modprobe.conf alias eth0 be2net alias eth1 be2net alias eth2 bnx2 alias eth3 bnx2 alias scsi_hostadapter cciss alias scsi_hostadapter1 usb-storage alias net-pf-10 off alias ipv6 off options ipv6 disable=1 alias bond0 bonding alias bond1 bonding cat ifcfg-bond0 DEVICE=bond0 IPADDR=10.201.21.124 NETMASK=255.255.255.192 NETWORK=10.201.21.64 BROADCAST=10.201.21.127 ONBOOT=yes BOOTPROTO=none USERCTL=no GATEWAY=10.201.21.65 TYPE=Ethernet BONDING_OPTS="mode=1 primary=eth0 arp_interval=200 arp_ip_target=10.201.21.65" ifcfg-eth0 # Emulex Corporation OneConnect 10Gb NIC (be3) DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none TYPE=Ethernet HWADDR="9C:8E:99:1F:05:40" cat ifcfg-eth1 # Emulex Corporation OneConnect 10Gb NIC (be3) DEVICE=eth1 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none TYPE=Ethernet HWADDR="9C:8E:99:1F:05:44"

Have you solved your issue.

Regards