topic Re: SCP is slow in 1 out of 3 directions!!! different than most things in Operating System - HP-UX

SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Fri, 04 Jan 2013 02:07:17 GMT

ok if i do a scp test1:/tmp/junk /tmp/junk it takes over 5 minutes from machine test2, if from test1 i run the same thing copying from test2 to test1 (scp test2:/tmp/junk /tmp/junk) it takes 5 seconds. We have verified /etc/hosts tables, DNS entries are correct.

We have been working with HP, Cisco, Epic and no one knows what is causing this? So i am sending out this help call from anyone with any ideas? ...

these are HP-UX blades, running 11vi3.

thanks for any ideas.

P.S. this thread has been moved from HP-UX>System Administration to HP-UX>networking - HP Forums moderator

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Bill Hassell — Thu, 03 Jan 2013 13:17:43 GMT

I would start with a verification of the ssh versions on each machine:

ssh -V

If they all match, check the KNRG (random number generator) module on each system:

swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN

(they should all match)

Also check both systems with top during the slow transfer period.

Does one of systems have exceedingly large amounts of CPU time?

Finally, make sure that name resolution is identical on both systems. Use nsquery rather than nslookup:

nsquery hosts systema
nsquery hosts systema.mydomain.com
nsquery hosts 10.11.12.13

nsquery hosts systemb
nsquery hosts systemb.mydomain.com
nsquery hosts 10.11.12.14

Be sure to query the remote AND the local host by name, fully qualified name and by IP address.

Re: SCP is slow in 1 out of 3 directions!!! Different than most things

Ken Grabowski — Thu, 03 Jan 2013 14:29:16 GMT

I have seen this happen when the routing tables are wrong. Often multiple routes in the table causing one host to try multiple paths with each packet sent. What does netstat -rn show on test1? Try a traceroute from test1 to test2 and see what it reports back.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 15:44:47 GMT

all of the ssh -v, swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN, and the nsquery's have worked fine so far, I am having my Networking team look into the routes...

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 18:46:47 GMT

we checked routing tables and stuff, i was told it is on the same layer 2, no hopping, tables all look good according to our head NE person.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 19:06:57 GMT

So what was the output from netstat -in and the results of traceroute from test1 to test2?

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:13:58 GMT

traceroutes only showed 1 jump, to the other machine, both directions...and is it a netstat -rn or netstat -in, the -rn showed routing tables, but the network engineers said they were fine, and if we go to one of the other new machines, that has this same problem with NO additional routes, it still fails.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 19:17:07 GMT

Sorry, netstat -rn.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:22:15 GMT

ok, the netstat -rn just shows the routes, correctly accoding to the Head Network Engineer, they match up, cluster packages are good, etc...

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 19:25:10 GMT

Marvin, if you dont post results then there is no point to asking for help here.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:30:23 GMT

ok not sure what you are wanting, but here is what it looks like...

khupepc001:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.54         172.27.104.54      UH    0    lan900    32808
172.27.111.246        172.27.111.246     UH    0    lan902    32808
172.27.104.215        172.27.104.215     UH    0    lan900:1 32808
172.27.103.214        172.27.103.214     UH    0    lan901    32808
172.27.103.215        172.27.103.215     UH    0    lan901:1 32808
172.27.111.240        172.27.111.246     U     2    lan902     1500
172.27.103.192        172.27.103.214     U     3    lan901     1500
172.27.103.192        172.27.103.215     U     3    lan901:1   1500
172.27.104.0          172.27.104.54      U     3    lan900     1500
172.27.104.0          172.27.104.215     U     3    lan900:1   1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

and on the other machine is

khepx019:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.37         172.27.104.37      UH    0    lan900    32808
172.27.103.207        172.27.103.207     UH    0    lan901    32808
172.27.104.0          172.27.104.37      U     2    lan900     1500
172.27.103.0          172.27.103.207     U     2    lan901     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

the problem is pulling a file from khupepc001 when you are logged into khepx019..

if this is not what you are asking please let me know, i am a little confused on this part.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:31:59 GMT

here is the netstat -rn from another system it still doesnt work from.

Routing tables
Destination           Gateway            Flags Refs Interface Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.55         172.27.104.55      UH    0    lan900    32808
172.27.111.247        172.27.111.247     UH    0    lan902    32808
172.27.103.212        172.27.103.212     UH    0    lan901    32808
172.27.111.240        172.27.111.247     U     2    lan902     1500
172.27.103.192        172.27.103.212     U     2    lan901     1500
172.27.104.0          172.27.104.55      U     2    lan900     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 19:42:29 GMT

which of these is test1? and which address do you aim at when doing the test that is slow?

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:44:35 GMT

we are logged into khepx019, and are doing a scp khupepc001:/tmp/aectst /tmp/aectst and it takes like 5 minutes, if i am logged into either of them and do a scp /tmp/aectst khupepc001:/tmp/aectst it takes 5 seconds.

and according to our EMR people this is a major problem.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:47:39 GMT

i guess the commands are really setup like this

if logged into khepx019 and isues a scp khupepc001:/tmp/aectst /tmp/aectst it takes 5 minutes

if logged into khepx019 and issues a scp /tmp/aectst khupepc001:/tmp/aectst it take 5 seconds

if logged into khupepc001 and issues a scp /tmp/aectst khepx019:/tmp/aectst it takes 5 seconds

hopefully this is a little clearer.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 19:48:02 GMT

From khepx019 try using individual IP addresses not host names and use each of the IP's that they have. See if there is any difference in the time it takes.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 19:53:22 GMT

logged into khepx019 doing a scp 172.27.104.54:/tmp/aectst /tmp/aectst takes the standard 5 minutes.

doing a push from khupepc001 scp /tmp/aectst 172.27.104.37:/tmp/aectst takes 5 seconds

doing a push from khepx019 scp /tmp/aectst 172.27.104.54:/tmp/aectst takes 5 seconds.

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 20:05:47 GMT

whats the time for:

khepx019 doing a scp 172.27.103.214:/tmp/aectst /tmp/aectst

Re: SCP is slow in 1 out of 3 directions!!! different than most things

marvin51796 — Thu, 03 Jan 2013 20:12:18 GMT

5 min

Re: SCP is slow in 1 out of 3 directions!!! different than most things

Ken Grabowski — Thu, 03 Jan 2013 20:12:57 GMT

Also, while the 5 minute scp is occurring, what is the connect on the remote system? netstat -an and look for the "ESTABLISHED" line from the originating server.