scp gives only 906.1KB/s between 2 data centers.

[Silver_1]

Gurus,

We have 2 data centers which are in different states and the second one just came up.
We have transferred one of our systems an RP8400 to the second Data Center.

Now i am trying to copy a 1Gb file bewteen two RP8400 but gets throughput of 1Mb/s. The scp happens over a WAN and there is no bandwidth issue.

Do you have any comments ?

Tx.

[Patrick Wallek]

What type of WAN connection do you have between the data centers?

Also check your network settings at the remote site and make sure that the settings for the NIC is correct. Especially check the duplex settings and verify that there isn't a duplex mismatch between the card in the server and switch port.

Mis-matched duplex can cause problems like you see.

[Silver_1]

Hi,

I have the similar settings on the NICs.

1000, NO NG, Full DUPLEX.

Also, we have other (OS) servers which can copy at a speed of 16mb/s.

Tx.

[Patrick Wallek]

OK, are you absolutely SURE that the NIC is actually communicating at 1Gb speed? Lots of the 1Gb NICs **require** that the NIC and the switch port be set to AUTO-NEG in order to get the maximum speed.

You might try the Auto negotiate settings and see what happens.

[Silver_1]

root@hpdd01:/root# lanadmin -x 10
Speed = 1000 Full-Duplex.
Autonegotiation = Off.

I have assurance from Net Team that the switchs are in 1000 FD off....

Tx

[Michael Selvesteen_2]

some thing to try

1. Using weak cipher for encrytion will definitely increase the throughput ( arcfour may be a good option)

2. You can also consider the steps available from the link

http://www.psc.edu/networking/projects/hpn-ssh/old-site/

All the best.

[Florian Heigl (new acc)]

Someday this will drive me mad:

100BaseT Ethernet, normal PC and normal Switch: autoneg
100BaseT Ethernet, HP-UX and Cisco: Hard Full Duplex settings
1000BaseT Ethernet, any variation: autoneg


With 1000BTX autonegotiation is not something optional. :)

[RAC_1]

As I understand and recall, the GB cards have to have autoneg on. On both ends of cards, do lanadmin -g "nm_id", if you have autoneg, problems, you will see fcs errors.

Also, set scp to "exchange of public/privat keys" authentication. What version of scp?? Old versins use, in-built random number generation and takes time in establshing a connection. Update to latest versions.

[Steve Lewis]

Temporarily enable ftp between the sites and compare the copy times with scp.

FTP is normally quicker than ssh/scp.

If the times are similar it points to a network problem.

If the ftp times are _much_ quicker and closer to your network speed, then it points to an ssh problem.

[Bill Hassell]

WAN is a VERY big issue. Generally, a WAN can be a simple UUCP network with a 2400 baud modem all the way to a SONET fiber link that run at gigabyte/sec. SO you have to start by characterizing the WAN speed. Is it a T1 link (1.144 Mbits) or a DSL link? Then you'll NEVER get more than 50Kb to 90Kb data transfer rates. Is it a T3 link (45Mbits), then you'll get 1-3 Mbyte/sec transfer rates. BUT ONLY if you use ftp.

The reason is that simple protocols like scp generate a massive number of bidirectional packets and that will kill your network performance. The problem is that the turnaround time from transmit to receive is usually very slow, from dozens to hundreds of milliseconds. You can see this with a ping test. Run it for a minute or so and see what the delay is. On a 10Mbit LAN, the time is usually 0-2ms. On a WAN, it might be 50ms to 500ms. Now the data packets might run at 4Mbytes/sec but unless the data packets are streamed without waiting for a status (like ftp), each record will require an acknowledgement and will incur the WAN turnaround time. During that time, there is dead air (nothing going on in either direction).

[Devesh Pant_1]

Nair,
I would suggest trying a normal test ftp instead of scp and check the statistics using netstat or nwadmin. You might also want your network team to have some sniffers turned on for the WAN

thanks
DP

[Ranjith_5]

Hi,

We are also facing the same problem. Because secure copy makes lot of encription during the file transfer and may slow down the data transfer.we are also getting only 1Mb/s max using secure copy between 2 data centers. finally tested the file transfer with normal FTP and found we are getting upto 12Mb/s.

Regards,
Syam

[Bill Hassell]

scp indeed will require compute power on both ends. This would only be a concern if you are running older equipment (less than 500Mhz processors). You can check the CPU load during the scp transfer. If it is more than 30% or so, the computation may indeed be slowing the transfer rate. It all cases, the ftp test will confirm the bottleneck.

[Steven E. Protter]

My approach to this issue would be to test a number of transfer methodologies.

ftp
scp
sftp

The encryption of scp is probably causing the slowdown.

Also, I wish to confirm that those GB cards must be set to autonegotiate.

You will under all circumstances get substantially less throughput than your band with due to encryption and other issues. The more CPU you have at both ends, the less of a problem this will be.

Its interesting that you are getting 1 Mb/s

A lot of Internet connections go 1.5 Mb/s and if this WAN is really a VPN over the public Internet the speeds you are getting are normal.

I'd be interested to see lanadmin -x output on the NIC cards involved.

SEP

[rick jones]

Things to check, in no particular order, some duplicates from other posts most likely:

1) Is _any_ CPU on _either_ side of the transfer "pegged" (ie saturated)? If so, then it is a case of crypto performance with scp and the crypto it is using

2) are there TCP retransmissions during the scp session. Do something like this:

netstat -p tcp > before; scp ... ; netstat -p tcp > after

and then get beforeafter from ftp.cup.hp.com dist/networking/tools/ and

beforeafter before after > delta

and look at delta

3) run a netperf TCP_STREAM test between the two systems. Vary the setting of -s and -S to see how large a socket buffer/TCP window you need to get link-rate on your WAN link

netperf -H -l 30 -- -m 4K -s 32K -S 32K

and try 64K, 128K, 256K etc until you see a peak.

www.netperf.org I prefer to use netperf over ftp because netperf "just" excercises the networking and not the filesystem.

The trouble could be that scp does not use a large enough window on top of the TCP window.
IIRC there have been some recentish changes in either ssh or scp, I cannot recall which, to let it put more data out onto a connection at one time.

[Silver_1]

In our environment, ftp is disabled on all the servers and are not allowed. So i don't have an option to enable ftp.

Tx,
Nair.

[rick jones]

be that as it may, how about the other stats folks have asked for as part of diagnosing what may be going-on?

[Silver_1]

Still the issue exists.

Working with vendor to get a software which can give a faster throughput over the WAN.