GB Speed -- scp Transfer -- Issue Getting > 40MB/s Transfer Rate

[Pawan_1]

We have 2 DL-585 Running Linux and have GB NIC Cards. Somehow when we transfer files between the 2 servers, we get max transfer rate of 40MB. Have tuned the following:
# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500


Any other way to increase the throughtput.....

[Ivan Ferreira]

The problem is that scp has overhead in the job of encrypting and decrypting the file. If you need better performance, consider using rsync, NFS, ftp or the old and "suppossed" insecure rcp.

[Hein van den Heuvel · Accepted solution]

Agree on "Secure" overhead in Scp.

Also, 40 MB/sec is not too bad, considering 100MB/sec is the absolute.

Finally, YOu did not mention the storage subsyste, are you sure that you can source and sink much more than 40MB/sec on you storage subsystem. If you are using just 1 disk then 40MB /sec is likely the limit.

Did you verify read and write speeds for a substantial file, say 1GB, with for example dd bs=128K count=8000 using /dev/zero or /dev/nul depending on the direction?

Regards,

Hein van den Heuvel
HvdH Performance Consulting.

[Vitaly Karasik_1]

agree, it's not too bad!

By checking top/iostat you can understand what is your bottleneck - CPU/NIC/disks.

I can recommend IBM's "Tuning IBM eservers xSeries Servers for Performance" (http://www.redbooks.ibm.com/abstracts/sg245287.html), especially chapter 8.

[rick jones]

Just because you've changed the sysctls doesn't necessarily mean a given application will use the larger values - it might be making its own setsockopt() calls to ask for a given socket buffer size. Strace against scp commands could confirm/deny that.

Also, scp has I believe its own flow-control - you want to make sure that it is configured (I'm not sure how) to actually take advantage of a large window.

Others have mentioned the discs - good to check - and that encryption has non-trivial overhead - for that one you want to look at the CPU utilization of all the individual cores rather than the average of the system. A single TCP connection (ie a single scp session) isn't going to get much from more than one or maybe a little of a second core - if you have a single core maxed-out on either end, you know what your bottleneck is.

Also, check for lost packets and retransmissions - ethtool and netstat stats from before and after the scp transfer, run them through beforeafter to see the numbers for the run. ftp://ftp.cup.hp.com/dist/networking/tools/

Finally (well nothing is ever final :) you might try some transfers without filesystem - say a netperf TCP_STREAM test - http://www.netperf.org/

[Pawan_1]

Thanks and the docs helped. The 40MB is the best transfer rate we got using scp and it also attributed to the i/o rates and cpu utilizations.

[Pawan_1]

As per the notes as suggested by various members.

[Florian Heigl (new acc)]

As this was mentioned nowhere - are you actually using jumbo frames?
most gigabit nics will peak out around 40MB/s otherwise, and will only reach higher speeds using jumbo frames.

I think only SysKonnect used to make GigE adapters that _never_ did under 85MB/s. But that was before they got bought out for their name, the newer adapters are based on marvell chips and no better what is on every gamers' motherboard.

[rick jones]

It has been _ages_ since anyone shipped a Gigabit ethernet interface which was unable to achieve link-rate with a 1500 byte MTU.

[Pawan_1]

How do you check if you are using Jumbo Frames ?

[rick jones]

A couple ways - one would be to look at the output of ifconfig, which under (most/all?) Linux distros reports the MTU. It is also how one changes the MTU.

However, you cannot just change the MTU of one interface in isolation. A switch to JumboFrames requires that _everything_ in the broadcast domain (all that which is connected with switches rather than routers) be capable of supporting, and be configured to use jumbo frames.

[Pawan_1]

Thanks and will try to work on the same. Will Update on the progress.