NIC Bonding under Redhat 4 on DL 585 G2

[JBARKER]

Hi all,
This is my first post here so apologies if I'm in the wrong area. I have an issue with a DL585 G2 I have recently built. I am trying to setup NIC bonding/teaming so that we have some resiliency on the server, I've set this up in the past and it's always worked fine. My kernel is at:
2.6.9-55.ELsmp x86_64

I started the build with the Firmware CD v7.80, I then installed Red Hat Enterprise Linux 4 AMD64. Did a full update from the Redhat network and then finally applied the Proliant Support Pack for Red Hat Enterprise Linux 4
version 7.80
The problem I now have is with bonding the two on-board NICs into one virtual. I have setup /etc/sysconfig/network-scripts/ifcfg-bond0, eth0 and eth1 as well as /etc/modprobe.conf in the usual way. Having set a ping running I normally test by pulling the patch cables individually and making sure that the ping continues via both NICs. My problem is this does not work on this server and I cannot figure out why.
I'm not sure of this is a hardware issue or a an OS issue, I suspect the latter.

Any help anyone can give me on this is most appreciated.

Thank you

Jon Barker

[Ivan Ferreira]

What is "does not work"?

Does not work at all, does not work "failover"?

Please post the output of:

ethtool eth0
ethtool eth1
ifconfig -a
netstat -ni
service network restart
tail /var/log/messages (after network restart)

I suggest you to add ethtool options to the interfaces:

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
ETHTOOL_OPTS="speed 100 duplex full autoneg off"
MASTER=bond0
SLAVE=yes

[jonnynoh]

Hi Ivan,
Yes the failover is not working sorry I wasn't specific.
The output is in the attachment.
Many thanks

[Ivan Ferreira]

Everything looks correct, what is the output of:

cat /proc/net/bonding/bond0

Before testing the failover and after testing failover.

[jonnynoh]

Output in attachment

I understood the NICs in the server to be Broadcom (bnx2) but as you can see it mentions NC371i also in the mail log, is there a difference?

Many thanks for your help so far

[Ivan Ferreira]

What I can see is that after the failover test, the NIC remains up. And another thing to note is that you receive a message that the NIC is down and then up again.

Jun 19 15:08:53 fldryolus01 kernel: bnx2: eth0 NIC Link is Down
Jun 19 15:09:48 fldryolus01 kernel: bnx2: eth0 NIC Link is Up, 100 Mbps full duplex, receive & transmit flow control ON

It seems that your NIC does not stay down.

[jonnynoh]

The second message is where we re-opened the switch port for eth0 and thus it comes back online.
What I've always seen in the past is if you pull the physical eth0 patch, eth1 should automatically take over and providing the switch port has portfast enabled you don't drop a single ping, i.e. active-backup.
The failover is not working because it appears eth1 never gets promoted to primary slave

[Steven E. Protter]

Shalom Jon,

I'd like to see modprobe.conf

Also ifcfg-eth0

I've got those cards bonding no problem in several servers.

There is a hardware check on your server boot cd. Its worth running.

SEP

[jonnynoh]

Hi SEP,

I'm glad you've managed to get these cards working as I've actually got 2 servers here with the exact same issue and both are DL585 G2s running the AMD64 Redaht 4 OS. Anyway output is in the attachments.
"There is a hardware check on your server boot cd. Its worth running."
Please tell me more, is this found during POST or on the Redhat CDs????

Regards
Jon

[Jim Healer]

So, I'm on pins and needles here. How did it work out?

Don't leave me hanging.

[PAVIC Thierry]

- 1 - Know your network card
lspci
...
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 11)
...
Know reference
(root:/)#lspci -n |grep "03:00.0"
03:00.0 Class 0200: 14e4:164c (rev 11)
(root:/)#cd /lib/modules/2.6.9-42.ELsmp
(root@:/lib/modules/2.6.9-42.ELsmp)#grep "14e4" modules.pcimap |grep "164c"
bnx2 0x000014e4 0x0000164c 0xffffffff 0xffffffff 0x00000000 0x00000000 0x0

Driver is --> bnx2

Update /etc/modprobe.conf file
alias eth0 bnx2
alias eth1 bnx2
alias scsi_hostadapter cciss
alias usb-controller uhci-hcd
alias usb-controller1 ehci-hcd
install bond0 /sbin/modprobe bonding -o bond0 miimon=100 mode=1

Check DNS /etc/resolv.conf
(root:/root)#more /etc/resolv.conf
domain MyDomain.net
nameserver XXX.XXX.XXX

Check bonding file
cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
USERCTL=no
ONBOOT=yes
BOOTPROTO=none
BROADCAST=XXX.XXX.255.255
IPADDR=XXX.XXX.XXX.XX
NETMASK=255.255.0.0
NETWORK=XXX.XXX.XXX
TYPE=Ethernet
IVP6INT=no
PEERDNS=yes
ETHTOOL_OPTS=speed 1000 duplex full autoneg off"

(root:/etc/sysconfig/network-scripts)#cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

(root:/etc/sysconfig/network-scripts)#cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Take care to the route
(root@linsm08:/etc/sysconfig/network-scripts)#cat /etc/sysconfig/network-scripts/route-bond0
GATEWAY1=XXX.XXX.XXX
NETMASK1=255.255.0.0
ADDRESS1=XX.XX.XX.XX
GATEWAY0=XXX.XXX.XXX
NETMASK0=255.255.0.0
ADDRESS0=XXX.XXX.XXX

File Copy (root:/etc/sysconfig/network-scripts)
#cp route-bond0 route-eth0
#cp route-bond0 route-eth1