Slow internet downloads with bond on RHEL5

[Chris House]

I have RHEL5 installed on a DL380 G5 with 2 NICs bonded together on bond0. The nics are connected to seperate switches which have an ISL between them, and the bond is configured in active-backup mode with one nic active and the other passive.

This works fine and failsover quickly if the active link goes down.

Using wget, I can download an ISO at about 11.1 MB/s from a local webserver on our network. However if I download an ISO from an internet server nearby (same town), it's dreadfully slow, around 4k/s. If I download same ISO on internet server from a different local server or my workstation, it goes very quickly like it should.

Our network team swears there is no rate limiting for this server at the firewall or anywhere. I changed the IP to a different one hoping it would make a difference, but it didn't.

Same issue occurs if the NICs are not bonded. Haven't tried different switch ports but network guys say those look fine and look like every other server port. No port errors.

Any ideas???

[rick jones]

a picked nit, but if the same issue happens when the NICs are not bonded, the subject is a triffle misleading :)

a quick look at tcp statistics might be interesting, as well as link-level stats via ethtool -S .

after that, packet tracing might be indicated to see the characteristics of the stream of traffic arriving at your system. being able to compare the stats and the trace with a "known good" system would be good.

comparing the ping times to reach the local server vs the Internet server would be good.

[Steven E. Protter]

Shalom,

Two ideas come to mind.
1) Network issues or intentional band width limitation on the local internet server. If its hosted in an office or DSL environment bandwidth for uploads would be severely limited.

2) A problem with this system, which seems unlikely.

SEP

[Vitaly Karasik_1]

are there any errors in "ifconfig" output?

do you have other RHEL into your local network?

[Chris House]

To rick - sorry for the misleading title, I was angling toward it being the problem but as I realized it probably wasn't, I didn't update the subject.

Also, ethtool -S eth0 or eth1 does not show any errors or collisions.

To steven - I was thinking bandwidth limitation by our network team but they swear up and down that they aren't limiting it either by its IP, mac address, or switch ports.

To Vitaly - bond0, eth0, and eth1 do not have any errors in ifconfig output.

I have a RHEL3 server on the network that DOES NOT exhibit this problem (neither do Windows servers). I checked several recent Fedora 8 servers and they DO show the same problem as this RHEL5 server (slow internet download but fast local download)

I have disabled iptables and ip6tables hoping that would make some difference even though i've never configured them; it did not help.

So it seems there's some misconfiguration on this RHEL5 install and the Fedora8 installs on other servers.

I'm testing by simply trying to fetch any large file off the internet, such as an ISO (wget http://www.kernel.org/pub/dist/knoppix-dvd/KNOPPIX_V4.0.2DVD-2005-09-23-EN.iso)

From a windows server on the same network, the speed is around 2.3MB/Sec which is what I would hope to get on the linux servers.

The RHEL3 server gets about 488K/s which is still better than the paltry 6.70K/s that the RHEL5 and Fedora8 servers get.

Traceroute to www.kernel.org is the same on the servers.

Here is my /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=10.10.105.64
NETMASK=255.255.248.0
NETWORK=10.10.104.0
BROADCAST=10.10.111.255
ONBOOT=yes
GATEWAY=10.10.104.1
USERCTL=no
BOOTPROTO=static
ONBOOT=yes

/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:1B:78:BB:FD:16
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes

/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:1B:78:BB:FD:14
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes

/etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias scsi_hostadapter cciss
alias scsi_hostadapter1 usb-storage
alias bond0 bonding
options bond0 mode=active-backup primary=eth0 miimon=100

[rick jones]

It is beginning to sound like time for some packet tracing.

[Vitaly Karasik_1 · Accepted solution]

Christopher,

It is "tcp window scaling issue" in kernel2.6

Try toupgrade to the latest kernel, recommendedby RH or deploy workaround from

http://inodes.org/blog/2006/09/06/tcp-window-scaling-and-kernel-2617/

[Chris House]

Vitaly, that totally did the trick for me. I set net.ipv4.tcp_window_scaling = 0 in /etc/sysctl.conf and rebooted, now I am enjoying 900+k/s instead of 6k/s on internet downloads. It's not the 2MB/s that I get with my vista laptop, but it's better than nothing.

Big thanks.

[rick jones]

Sounds though you may still want to go and shoot a firewall in the head. Given how long window scaling has been out there in the TCP specs, that there is still active kit not working correctly with it is a very bad thing...