DL360 G3+RedHat 9.0 bcm5700 problem

[S.H.Lan]

I have serveral DL360 server and all of them are installed Redhat 9.0.
The default network module is tg3 and I replaced it to bcm5700-7.1.9c-1.
I install bonding module after bcm5700 load finish.
And then I edit the ifcfg-bond0, ifcfg-eth0, ifcfg-eth1 .
I am sure that I make all step right cause I do all most of them successful.
But just only one server of them can't work.
I saw the boot.log under /var/log and it's showing :
Mar 24 14:56:48 pt-web01 network: Setting network parameters: succeeded
Mar 24 14:56:48 pt-web01 network: Bringing up loopback interface: succeeded
Mar 24 14:56:48 pt-web01 network: Bringing up interface eth0: succeeded
Mar 24 14:56:48 pt-web01 network: Bringing up interface eth1: succeeded

But when I run ifconfig command I just can see lo device.
I can't fix the problem so I rmmod bcm5700 and insmod tg3 again and edit all the ifcfg-eth* to the standard file. And then I reboot it.

But the networking still can't work!!
I try to use "ifconfig" command to manu configure eth0 and eth1 as follow:
# ifconfig eth0 10.0.1.1 netmask 255.255.255.0
# ifconfig eth1 192.168.1.1 netmask 255.255.255.0

and they can work....


Why?????????????

[Bill Wood_3]

Does your modules.conf and ifcfg-xxx files look similar to the following?

modules.conf:

alias eth0 bcm5700
alias eth1 bcm5700
alias bond0 bonding
options bond0 miimon=100 mode=0

ifcfg-bond0:

DEVICE=bond0
IPADDR X.X.X.X
NETMASK X.X.X.X
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

ifcfg-eth0:

DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

ifcfg-eth1:

DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Please post the contents of these files if not.

[S.H.Lan]

Yes, I have edit ifcfg-bond0, ifcfg-eth0, ifcfg-eth1 and modules.conf as your posting.

Actually, I have 10 DL360, 4 for Redhat AS 2.1, and 6 for RedHat 9.0.

I did all of them for bonding, and just 1 server can't work well..
I can't see eth0 and eth1 when I run ifconfig command.

I try to recover ifcfg-eth0 ,ifcfg-eth1 and modules.conf to original mode (set ip and no master, slave parameter ) and delete ifcfg-bond0 file for testing. But networking still can't work well... I must load bcm5700 module , assign IP and up NIC by myself.

I can't understand, why just 1 server can't work?? All of the environment are the same...

[Stuart Browne]

Show us the contents of /etc/modules.conf please, as well as the output of the 'lsmod' command.

Also show us any output when you type "service network restart".

[S.H.Lan]

hello all, I've found the problem and made the bonding working now.

But I still have other question.

Q1: Set the bonding mode=0 (RR mode)
ping the default gateway and unplug eth1 cable. I can see the eth1 lind down message on console and ping to gateway is still alive.
And then I plug the eth1 cable back and unplug eth0 cable, the ping is pause.
I restart the ping command and just saw the network unreachable message.

Q2: Set the bonding mode=1 (Active-backup mode)
the same steps as above and the same result as above, too.

I can't unplug the eth0 cable, it makes the network unreachable :(

Is any good suggestion to set the bonding?

[Jeroen Peereboom]

I experience the same.
Since I have both NICs attached to 1 switch I didn't test RR mode, see bonding.txt file).
See my question http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=517376

My explanation is that some traffic going from the server to the switch is needed to keep the switch informed of the mac address attached to the switch ports. If I pull out the cable, it still tries to send traffic to the disconnected port.
I don't know if that explains the RR situation too.

JP

P.S. By the way, how did you solve your problem?

[Bill Wood_3]

How are you nics connected to the network?

[Bill Wood_3]

How are your nics connected to the network?

[Jeroen Peereboom]

I should have added that both NICs are presented by the bonding module to the switch with 1 mac address...

If you have the NICs connected to separate switches, this should not occur.

JP.

[S.H.Lan]

Well, I use 2 layer-2 switch.
eth0 is connecting to switch-1 and eth1 is connecting to switch-2.

And now, I got a information about another problem now :(

====
Environment:
Network device--
1)Core switch for VLAN: 10.0.1 and 10.0.2
2)Each VLAN has 2 switch connect to Core switch for HA

Linux Server (server1, 10.0.1.x)
1)Redhat 9.0
2)2 NIC, eth0 and eth1, eth0 -- switch1, eth1 -- switch2
3)binding to be a virtual NIC by bonding mode=0

Windows Server (server2, 10.0.2.x)
1)Windows 2003
2)2 NIC, NIC1 -- Switch1, NIC2 -- Switch2. bind to be a virtual NIC by HP Network Utility (Fault Tolerence)

====
Situation:
1) and the cable are connecting to switch
ping server1 from server2
Reply from 10.0.1.x: bytes=xx timeReply from 10.0.1.x: bytes=xx timeReply from 10.0.1.x: bytes=xx timeRequest timed out.
Reply from 10.0.1.x: bytes=xx timeReply from 10.0.1.x: bytes=xx time
The packets is lost!

2)unplug cable from eth1 on server1
ping from server2 again.

packet lost=0

====
For making the servers connect to each other working well, I have to unplug the cable from eth1 now :(

Q1: Should I use mode 0 (RR mode) or mode 1 (Active-backup)?
Q2: Any good solution about this situation?


------------
Sorry for my poor English to explain it well :((

[Jeroen Peereboom]

1. Mode 1 works more often than mode 0. Mode 0 needs trunking on the switches.

2. I do not understand if you plugged a cable causing the timeouts.

You have complicated the setup by having 2 servers with network teaming / bonding in stead of only the Linux server.

Maybe it's a good question for the networking forum?

JP.