slow scp

[Steve Coates]

I am having an issue with one of my systems. When I do an scp to or from the machine I am only getting 500-600KB/sec. On my other systems I see closer to 7-8MB/sec.

Patches are consistant across machines, all are running 11.11, all are connected with GB network.
OpenSHH 3.6.1p2. Network settings on the machines are consistant.

I have checked the ssh_config and sshd_config and they are the same on all systems.

[Jeff Schussele]

Hi Steve,

Check the network speed/duplex on that NIC with:

lanadmin -x Y

where Y=ppa# of the NIC - EX 1 for lan1 2 for lan2 etc.

If it's running 1/2 duplex this could explain the performance trouble. Then both the NIC and switch would need to be locked into full duplex.

HTH,
Jeff

[Steve Coates]

It is set to 1000 full auto

[RAC_1]

Apart from checking nic speed and settings (though if they are GB, they have to have at 1000 and autoneg on. they only do autoneg), you may want to check "how you have set up scp-have you set it up to work based on key exchange, password or host based authentication"

Also old ssh versions have problem with random number generation. They use different command to do that. If you delete some commands used for random number generation, you may be faster. Also up grading it to latest version is always better.

Anil

[Steve Coates]

The ssh and sshd config files are using default settings on the problem machine as well as the machines that are working fine.

I can not find any difference between the "good" and "bad" machine.

[RAC_1]

Steve,

You did not read the posting completely.
How you have configured scp?? What does it use-password, host or key bases authtication??

What did lanadmin -x "ppid_of_nic" tell you??

Also, if possible run rcp and see if it faster or not. That we can can at least isolate the problem.

Anil

[Steve Coates]

Anil,
I guess I am not sure what you are asking. It uses password and keys if I am understanding you.

The NIC is at 1000 full. I can not run rcp, it is not allowed in our enviroment.

[Gopi Sekar]

few more things which can generally cause this problem:

invalid/incorrect router and DNS information

do a ping to a remote machine and check the round trip machine it should be very very minimal.

and just for a confirmation try running scp with -C option it will enable compression on both ends, see whether this really makes any improvement on the transfer

Hope this helps,
Gopi

[PVR]

Hi Steve

If our machines are in the same network, pl check ping response from two workig machine and one with the system which is creating th e problem.

Also check your dns settings.

Please take one working machine ad the system creting problem and try putting the IP address for them in the /etc/hosts file and check scp bw the two.

If possile try rcp (I know it is dificult due to secuity reasons)and check the timings

[Steve Coates]

routes are good, ping is good and compression is enabled so I don't think it is a factor.

[Gopi Sekar]

mm, interesting... check for possible IP address conflict and also if the problematic machine has two ethernet controllers, try to do scp using the second one. For all you know it could be a hardware issue

[Steve Coates]

I only have one network connection at the moment. I did do a ftp (get) from the machine and it was as fast as I would expect. I think that rules out the network and puts it back on ssh/scp.

[Volker Borowski]

Hello,

how big is the file you use for testing ?
Since 600K to 7MB is roughly a factor of 10 I assume for sure the network is hooked up with 100MBit.
May be a segment in between has only fast ethernet connection or likewise.
May be a sitch autonegotiates down, but does not tell the adapter correctly.

I would proceed like this:

Tracroute from source to target
Tracroute from target to source

All hops on this path identical ?
Check network hookup on all hops.

Check the switches involved, by logging into the switch and examine portconfiguration of all ports involved in the route.

Hope this helps
Volker

[Steve Coates]

traceroutes are the same both ways. Ports are set up right.

[rick jones]

Definitely check the lanadmin and netstat stats. As for duplex, half-duplex in and of itself isn't necessarily bad, only if the other side is not also in half-duplex:

How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

Auto Half Full

Auto Happiness Lucky Sorrow

Half Lucky Happiness Sorrow

Full Sorrow Sorrow Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly :) Sorrow means that there _will_ be a duplex
mis-match.

When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of late collisions. On
the side running full-duplex you will see things like FCS errors.
Note that those errors are not necessarily conclusive, they are simply
indicators.

And, as pointed-out, GB _requires_ autoneg anyway.

Unless the network is otherwise loaded, ping will "never" see a duplex mismatch because it never has more than one packet on the network at one time, and it takes simultaneous access of the network to experience a problem with duplex mismatch.

Checking the CPU util on each end of the slow scp would be a good idea. Also, check the TCP window sizes by running a tcpdump and looking at the SYN segment to get the window scaling factor if any and then the window size.