Getting lot request time out in teamed ethernet card in RHEL5

[senthil_kumar_1]

Hi All,

There are four NICs available.

I have teamed those four NICs into one.

1)Ifconfig output:

# ifconfig
bond0 Link encap:Ethernet HWaddr 00:26:55:35:12:34
inet addr:10.247.0.17 Bcast:10.247.3.255 Mask:255.255.252.0
inet6 addr: fe80::226:55ff:fe35:1234/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:2768290243 errors:0 dropped:1391541 overruns:0 frame:0
TX packets:813877617 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4012421741416 (3.6 TiB) TX bytes:57452400530 (53.5 GiB)

eth0 Link encap:Ethernet HWaddr 00:26:55:35:12:34
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:208277598 errors:0 dropped:1391541 overruns:0 frame:0
TX packets:461670028 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:301278876409 (280.5 GiB) TX bytes:30385741745 (28.2 GiB)
Interrupt:169 Memory:f4000000-f4012100

eth1 Link encap:Ethernet HWaddr 00:26:55:35:12:36
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:151581155 errors:0 dropped:0 overruns:0 frame:0
TX packets:209205143 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:217232048639 (202.3 GiB) TX bytes:14637443757 (13.6 GiB)
Interrupt:154 Memory:f2000000-f2012100

eth2 Link encap:Ethernet HWaddr 00:26:55:35:12:38
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:920947066 errors:0 dropped:0 overruns:0 frame:0
TX packets:95181590 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1339304026454 (1.2 TiB) TX bytes:7322928273 (6.8 GiB)
Interrupt:194 Memory:f8000000-f8012100

eth3 Link encap:Ethernet HWaddr 00:26:55:35:12:3A
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1487484424 errors:0 dropped:0 overruns:0 frame:0
TX packets:47820856 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2154606789914 (1.9 TiB) TX bytes:5106286755 (4.7 GiB)
Interrupt:202 Memory:f6000000-f6012100



2)Configuration file of bond0:

# cd /etc/sysconfig/network-scripts

# more ifcfg-bond0
DEVICE=bond0
IPADDR=10.247.0.17
NETWORK=10.247.0.0
NETMASK=255.255.252.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

3)Configuration file of eth0:

# more ifcfg-eth0
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none


4)Configuration file of eth1:

# more ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

5)Configuration file of eth2:

# more ifcfg-eth2
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth2
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

6)Configuration file of eth3:

# more ifcfg-eth3
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth3
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none


My Issue:

Getting

My questions:

1)I need load-balance and fail-over, so that I have configured like above, Are above configurations correct?

2)Why it is happening?

3)How to resolve?

[Steven E. Protter]

Shalom,

1 more bit of information is needed:

/etc/modprobe.conf

Please post. This includes the mode of bonding and is important to resolving the issue.

Your questions:
1)I need load-balance and fail-over, so that I have configured like above, Are above configurations correct?

The above configurations look correct.


2)Why it is happening?

Possible these cards are mismatched and don't play well together. What is the type of card here?

3)How to resolve?

Possibly change the mode,check the switch configuration, it may not be compatible with the mode you are using. I would do a eyeball reality check and make sure all the lights are lit and the wiring is good.


Also, the statistics on the cards show an enormous amount of data is being moved and its not balancing well on the cards. That points to a mode change possibly being beneficial.


Please post the additional information and I'll try and deal with the details and provide further help.

SEP

[senthil_kumar_1]

Hi Steven,

1)Configured ethernet cards and drivers:

#lspci
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)


2)Output of /etc/modprobe.conf

alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2
alias eth3 bnx2

3)Output of ethtool command:

3.1)For eth0:

# ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes



3.2)For eth1:

#ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes


3.3)For eth2:

# ethtool eth2
Settings for eth2:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes


3.4)For eth3:


# ethtool eth3
Settings for eth3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Link detected: yes


My questions:

1)What do you want to verify in swith, please explain me in detail?

[Steven E. Protter]

Shalom,

From what I see in modprobe.conf bonding is not configured.

http://www.cyberciti.biz/tips/linux-bond-or-team-multiple-network-interfaces-nic-into-single-interface.html

http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding

Good Centos 5 thread which applies to rhel5

http://www.centos.org/modules/newbb/viewtopic.php?topic_id=10641&forum=40

A scrape:

# First line here is crticial
modprobe bonding mode=5 miimon=100
ifconfig bond0 80.229.71.221 netmask 255.255.255.0 up
ifenslave bond0 eth0 eth1


to understand bonding look at it as simple as this , when you create an alias interface eth0:0 for example
you create the interface as a slave to eth0 . all traffic from this slave will run from eth0. with bonding its the same
you create real interfaces as slave to virtual interface, all traffic will go from this virtual (bond)
to check the bond status

cat /proc/net/bonding/bond0


SEP

[senthil_kumar_1]

Hi Steven,

Sorry I forget to incluede followings in my above reply in the output of /etc/modprobe.conf.

So actually my /etc/modprobe.conf contains followings.

# more /etc/modprobe.conf

alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2
alias eth3 bnx2
alias bond0 bonding
options bond0 mode=balance-alb miimon=100


Please find the output of "/proc/net/bonding/bond0":

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:35:12:34

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:35:12:36

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:35:12:38

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:26:55:35:12:3a



Are my above configurations correct?

[Steven E. Protter · Accepted solution]

Shalom again,

modprobe looks good.

service network restart

Problem should be resolved. I'm not familiar with your chosen mode, but it meets the criteria you laid down earlier in this thread.

We are at the stage where you need to give it a try.

SEP