Two separate NIC bonding on one DL360 G4P RHEL4

[Kris Payne_1]

Hey everyone, I am stumped so I thought I'd see if I could throw this out to you all. I've got an NC77xx dual-port adapter and an NC71xx PCI-X dual-port adapter in there as well. When I setup bonding for the onboard card, everything worked perfectly, eth0 and eth1 slaved themselves to bond0 just fine and all was well. However, when I tried to create another bond (bond5) in the same fashion, it doesn't work and half the time eth1 won't come up in bond0 when I reboot unless I comment out the bond5 lines. Attached are my configuration files for the interfaces as well as modprobe. What am I missing? I couldn't find anything online to help me. doing ifup on both eth2 and 3 both give the 'slaving to bond5' message, but whenever I try to ifup bond5 it says 'bonding device bond5 does not seem to be present, delaying initialization.' I saw a suggestion somewhere to make its name zbond5 but that didn't do any good either. Thanks in advance for any help you sling my way =).

Kris

[Craig Gilmore]

Hi Kris,

I've never been able to get more than 1 bond device working on a system. I run into the same issues you detail.

I'm presuming that the bond driver and the kernel have some conflicts. I haven't taken the time to read through the source and see if my presumption of only 1 bond per system is justified.

Regards,

CG

[Anthony Martin_1]

Hi Kris,
Have you used the "max_bonds=" option in you modprobe.conf. Your line should look like this
options bond0 miimon=100 mode=0 max_bonds=3

would create bond0 bond1 bond2

Create your ifcfg-bond1 with the network info (normally I just copy ifcfg-bond0 and modify the network address).
Change your ethernet interface eg ifcfg-eth3 to use the new bonded interface "MASTER=bond1", restart the network (easiest way) and in theory, it should just all work.

I also have an alternative suggestion, but you should see how this goes first.

Hope this helps
Anthony

[Kris Payne_1]

Excellent, max_bonds did indeed solve the issue. Thank you !

[Scott Farwell]

The "max_bonds=" parameter is something new fro RHEL4. Redhat 7.3 and RHEL3.x do not require it. I have all 3 types running.

To get 2 bonds (192.168.2.145 & 10.0.0.1)working in an eth0-eth2 & eth1-eth3 configuration so each bond uses an internal NIC port and an external add-in NIC port in case the add-in card fails, these are the following configurations that are needed. This also works for RHEL3 & Redhat 7.3 without the "max_bonds=" paramter in the /etc/modules.conf.

/etc/modprobe.conf
----------------------
# The following lines have to be added to support 2 bonding interfaces.
alias bond0 bonding
alias bond1 bonding
options bond0 miimon=100 mode=1 max_bonds=2
options bond1 -o bonding1 miimon=100 mode=1

/etc/sysconfig/network-scripts/ifcfg-bond0
-------------------------------------------
DEVICE=bond0
BOOTPROTO=none
BROADCAST=192.168.2.255
IPADDR=192.168.2.145
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
TYPE=Ethernet

/etc/sysconfig/network-scripts/ifcfg-bond1
-------------------------------------------
DEVICE=bond1
BOOTPROTO=none
BROADCAST=10.0.0.255
IPADDR=10.0.0.1
NETMASK=255.255.255.0
NETWORK=10.0.0.0
ONBOOT=yes
TYPE=Ethernet

/etc/sysconfig/network-scripts/ifcfg-eth0
-------------------------------------------DEVICE=eth0
USERCTL=no
MASTER=bond0
ONBOOT=no
SLAVE=yes
BOOTPROTO=none

/etc/sysconfig/network-scripts/ifcfg-eth1
-------------------------------------------DEVICE=eth1
USERCTL=no
MASTER=bond1
ONBOOT=no
SLAVE=yes
BOOTPROTO=none

/etc/sysconfig/network-scripts/ifcfg-eth2
-------------------------------------------DEVICE=eth2
USERCTL=no
MASTER=bond0
ONBOOT=no
SLAVE=yes
BOOTPROTO=none

/etc/sysconfig/network-scripts/ifcfg-eth3
-------------------------------------------DEVICE=eth3
USERCTL=no
MASTER=bond1
ONBOOT=no
SLAVE=yes
BOOTPROTO=none

Set the default gateway in the /etc/sysconfig/network file along with your FQDN.

[Scott Farwell]

Device persistence can also be enabled to ensure that the NICs identifying themselves as eth1, eth2, etc... always remain on the same hardware ports in case of a failure of a single NIC port. You don't want your eth names to shift.

Upgrading to udev-095 from udev-039 that ships with RHEL4 is the smoothest solution, but that wasn't an option for me. Using names other than eth0 - eth3 also wasn't an option for me. Here is what we ended up using to get around udev-039's inability to re-use eth0-ethx names.

Create a udev rule using YOUR MACS
Create an /etc/mactab file using YOUR MACS
Modify /etc/init.d/network to run nameif

/etc/udev/rules.d/20-net.rules
--------------------------------
KERNEL="eth*", SYSFS{address}="00:0b:cd:69:c3:66", NAME="NIC1"
KERNEL="eth*", SYSFS{address}="00:0b:cd:69:c3:65", NAME="NIC2"
KERNEL="eth*", SYSFS{address}="00:11:0a:17:66:26", NAME="NIC3"
KERNEL="eth*", SYSFS{address}="00:11:0a:17:66:27", NAME="NIC4"

/etc/mactab
-------------
eth0 00:0b:cd:69:c3:66
eth1 00:0b:cd:69:c3:65
eth2 00:11:0a:17:66:26
eth3 00:11:0a:17:66:27

/etc/init.d/network
------------------------
(add right after
>>
# Check that networking is up.
[ "${NETWORKING}" = "no" ] && exit 0
<<)

# RDD: add 'nameif' usage; uses /etc/mactab
nameif || echo "nameif: reports error"