SCP is slow in 1 out of 3 directions!!! different than most things

[marvin51796]

ok if i do a scp test1:/tmp/junk /tmp/junk it takes over 5 minutes from machine test2, if from test1 i run the same thing copying from test2 to test1 (scp test2:/tmp/junk /tmp/junk)  it takes 5 seconds. We have verified /etc/hosts tables, DNS entries are correct.

We have been working with HP, Cisco, Epic  and no one knows what is causing this? So i am sending out this help call from anyone with any ideas? ...

these are HP-UX blades, running 11vi3.

thanks for any ideas.

P.S. this thread has been moved from HP-UX>System Administration to HP-UX>networking - HP Forums moderator

[Bill Hassell]

I would start with a verification of the ssh versions on each machine:

ssh -V

If they all match, check the KNRG (random number generator) module on each system:

swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN

(they should all match)

Also check both systems with top during the slow transfer period.

Does one of systems have exceedingly large amounts of CPU time?

Finally, make sure that name resolution is identical on both systems. Use nsquery rather than nslookup:

nsquery hosts systema
nsquery hosts systema.mydomain.com
nsquery hosts 10.11.12.13

nsquery hosts systemb
nsquery hosts systemb.mydomain.com
nsquery hosts 10.11.12.14

Be sure to query the remote AND the local host by name, fully qualified name and by IP address.

[Ken Grabowski]

I have seen this happen when the routing tables are wrong. Often multiple routes in the table causing one host to try multiple paths with each packet sent. What does netstat -rn show on test1?  Try a traceroute from test1 to test2 and see what it reports back.

[marvin51796]

all of the ssh -v, swlist -a readme -a mod_time -a revision -a software_spec -l fileset OS-Core.RNG-DKRN, and the nsquery's have worked fine so far, I am having my Networking team look into the routes...

[marvin51796]

we checked routing tables and stuff, i was told it is on the same layer 2, no hopping, tables all look good according to our head NE person.

[Ken Grabowski]

So what was the output from netstat -in and the results of traceroute from test1 to test2?

[marvin51796]

traceroutes only showed 1 jump, to the other machine, both directions...and is it a netstat -rn or netstat -in, the -rn showed routing tables, but the network engineers said they were fine, and if we go to one of the other new machines, that has this same problem with NO additional routes, it still fails.

[Ken Grabowski]

Sorry, netstat -rn.

[marvin51796]

ok, the netstat -rn just shows the routes, correctly accoding to the Head Network Engineer, they match up, cluster packages are good, etc...

[Ken Grabowski]

Marvin, if you dont post results then there is no point to asking for help here.

[marvin51796]

ok not sure what you are wanting, but here is what it looks like...

khupepc001:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.54         172.27.104.54      UH    0    lan900    32808
172.27.111.246        172.27.111.246     UH    0    lan902    32808
172.27.104.215        172.27.104.215     UH    0    lan900:1  32808
172.27.103.214        172.27.103.214     UH    0    lan901    32808
172.27.103.215        172.27.103.215     UH    0    lan901:1  32808
172.27.111.240        172.27.111.246     U     2    lan902     1500
172.27.103.192        172.27.103.214     U     3    lan901     1500
172.27.103.192        172.27.103.215     U     3    lan901:1   1500
172.27.104.0          172.27.104.54      U     3    lan900     1500
172.27.104.0          172.27.104.215     U     3    lan900:1   1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

and on the other machine is

khepx019:/# netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.37         172.27.104.37      UH    0    lan900    32808
172.27.103.207        172.27.103.207     UH    0    lan901    32808
172.27.104.0          172.27.104.37      U     2    lan900     1500
172.27.103.0          172.27.103.207     U     2    lan901     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

the problem is pulling a file from khupepc001 when you are logged into khepx019..

if this is not what you are asking please let me know, i am a little confused on this part.

[marvin51796]

here is the netstat -rn from another system it still doesnt work from.

Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.27.104.55         172.27.104.55      UH    0    lan900    32808
172.27.111.247        172.27.111.247     UH    0    lan902    32808
172.27.103.212        172.27.103.212     UH    0    lan901    32808
172.27.111.240        172.27.111.247     U     2    lan902     1500
172.27.103.192        172.27.103.212     U     2    lan901     1500
172.27.104.0          172.27.104.55      U     2    lan900     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.27.104.1       UG    0    lan900     1500

[Ken Grabowski]

which of these is test1? and which address do you aim at when doing the test that is slow? 

[marvin51796]

we are logged into khepx019, and are doing a scp khupepc001:/tmp/aectst /tmp/aectst and it takes like 5 minutes, if i am logged into either of them and do a scp /tmp/aectst khupepc001:/tmp/aectst  it takes 5 seconds.

and according to our EMR people this is a major problem.

[marvin51796]

i guess the commands are really setup like this

if logged into khepx019 and isues a scp khupepc001:/tmp/aectst /tmp/aectst it takes 5 minutes

if logged into khepx019 and issues a scp /tmp/aectst khupepc001:/tmp/aectst it take 5 seconds

if logged into khupepc001 and issues a scp /tmp/aectst khepx019:/tmp/aectst it takes 5 seconds

hopefully this is a little clearer.

[Ken Grabowski]

From  khepx019 try using individual IP addresses not host names and use each of the IP's that they have. See if there is any difference in the time it takes.

[marvin51796]

logged into khepx019 doing a scp 172.27.104.54:/tmp/aectst /tmp/aectst takes the standard 5 minutes.

doing a push from khupepc001 scp /tmp/aectst 172.27.104.37:/tmp/aectst takes 5 seconds

doing a push from khepx019 scp /tmp/aectst 172.27.104.54:/tmp/aectst takes 5 seconds.

[Ken Grabowski]

whats the time for:

 khepx019 doing a scp 172.27.103.214:/tmp/aectst /tmp/aectst

[marvin51796]

5 min

[Ken Grabowski]

Also, while the 5 minute scp is occurring, what is the connect on the remote system?  netstat -an and look for the "ESTABLISHED" line from the originating server. 

[marvin51796]

Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp        0      0  127.0.0.1.49163        *.*                     LISTEN
tcp        0      0  *.135                  *.*                     LISTEN
tcp        0      0  *.50006                *.*                     LISTEN
tcp        0      0  *.49157                *.*                     LISTEN
tcp        0      0  127.0.0.1.49154        *.*                     LISTEN
tcp        0      0  *.515                  *.*                     LISTEN
tcp        0      0  *.4045                 *.*                     LISTEN
tcp        0      0  *.49159                *.*                     LISTEN
tcp        0      0  127.0.0.1.49163        127.0.0.1.706           ESTABLISHED
tcp        0     48  172.27.104.37.22       172.27.104.54.61319     ESTABLISHED
tcp        0      0  127.0.0.1.706          127.0.0.1.49163         ESTABLISHED
tcp        0      0  127.0.0.1.7161         *.*                     LISTEN
tcp        0      0  *.49166                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.27.104.52.50896     FIN_WAIT_2
tcp        0      0  *.49160                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.27.104.247.55537    ESTABLISHED
tcp        0      0  *.22                   *.*                     LISTEN
tcp        0      0  172.27.103.207.52972   172.27.103.214.22       ESTABLISHED
tcp        0      0  172.27.104.37.22       172.25.114.131.54409    ESTABLISHED
tcp        0      0  *.901                  *.*                     LISTEN
tcp        0      0  *.7815                 *.*                     LISTEN
tcp        0      0  *.587                  *.*                     LISTEN
tcp        0      0  *.6112                 *.*                     LISTEN
tcp        0      0  *.5303                 *.*                     LISTEN
tcp        0      0  *.25                   *.*                     LISTEN
tcp        0      0  *.49158                *.*                     LISTEN
tcp        0      0  172.27.104.37.10443    172.26.104.52.60958     ESTABLISHED
tcp        0      0  *.111                  *.*                     LISTEN
tcp        0      0  172.27.104.37.49156    172.27.16.122.389       ESTABLISHED
tcp        0      0  *.2121                 *.*                     LISTEN
tcp        0      0  *.6389                 *.*                     LISTEN
tcp        0      0  *.2049                 *.*                     LISTEN
tcp        0      0  172.27.104.37.52979    172.27.104.37.1712      TIME_WAIT
tcp        0      0  *.5989                 *.*                     LISTEN
t\

[Ken Grabowski]

What catches my attention is the scp from khepx019 appears to create the following connection:

tcp        0      0  172.27.103.207.52972   172.27.103.214.22       ESTABLISHED

While khepx019 has:

172.27.103.207        172.27.103.207     UH    0    lan901    32808
172.27.103.0          172.27.103.207     U     2    lan901     1500

khupepc001 has:

172.27.103.214        172.27.103.214     UH    0    lan901    32808
172.27.103.192        172.27.103.214     U     3    lan901     1500

That would seem to indicate that your APA pseudo interface 901 on khupepc001 is trying to route traffic to a specific host, unless 172.27.103.192 is a different router.

I notice that on both non-khepx019 hosts your 900 based interfaces use a standard self and subnet route but the pseudo 901 and 902 interfaces appear to route to specific hosts.  These may be proper configurations, but since they are different from default configurations, and different from khepx019, this is where I would focus for now. I see nothing else out of the ordinary.

APA configurations can be rather complex, and I would hesitate to tell you to change something here without a lot of testing.  I would suggest going over the  /etc/rc.config.d/netconf,  /etc/rc.config.d/hp_apaconf, /etc/rc.config.d/hp_apaportconf, and if used /etc/lanmon/lanconfig.ascii file with APA experts at HP support.

[Dennis Handly]

>if logged into khupepc001 and issues a scp /tmp/aectst khepx019:/tmp/aectst it takes 5 seconds

To be complete, can you reverse this and get the 4th direction and time?

[Matti_Kurkela]

Modern blade servers should all have gigabit or better NICs. But are all the network links between khupepc001 and khepx019 gigabit, or are there any 100 Mbps hops? If there are, check for duplex mismatches on those hops. On 100 Mbps ethernet, disabling autonegotiation typically meant the NIC would completely stop sending autonegotiation messages and reacting to them. This led to duplex mismatches if one end of a link was forced to Full-Duplex and the other was on autonegotiation, since in the absence of autonegotiation messages from the other end, the autonegotiation logic must assume Half-Duplex. But on gigabit ethernet, "disabling autonegotiation" works a little differently: the NIC will still send autonegotiation messages, but will only offer a single option so there is nothing to choose. Some gigabit NIC chips will use the new disabling method on slower speeds too, others won't. The autonegotiation is a mandatory part of the gigabit Ethernet specification on copper cables, so it cannot be completely disabled in gigabit mode. To check the speed and duplex settings on a physical lanN interface on HP-UX, run "lanadmin -x N" where N is the number of the lanN interface (the number only: leave out the "lan" prefix).

[marvin51796]

Well just an update for everyone, to keep you in the loop. We have talked to Cisco and apparently they had us open up some buffers to max, and this cut the time in half down to 2 1/2 to 3 minutes....but apparently going from HP to Cisco equipment there is a problem with it handling burst copy. No one is sure what we should do now, but Cisco is looking into it......

Side not we did check all of the Duplex issues the first night this happened, and MTU settings , and setups on the OA's....WE, meaning HP, Epic didnt see any missmatches.