scp / sftp / even FTP stalled between hpux.

[PatRoy]

G'Day.

Here's the scenario.

Got 2 HPUX 11.31 servers, in different VLANs.

Server A as 10.1.1.47
Server B as 10.2.156.46

Now, I'm able to SSH from A to B without any problems. Ping works just fine! However, I am unable to transfer any files using either SCP, SFTP or even plain FTP! It just... stalls, either I try: A => B or B => A. Same results!

Output of scp (using -v verbose):

debug1: Authentication succeeded (keyboard-interactive).
debug1: Final hpn_buffer_size = 131072
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending command: scp -v -t /home/proy/
Sending file modes: C0666 747520 less-418-ia64-11.31.depot
Sink: C0666 747520 less-418-ia64-11.31.depot
less-418-ia64-11.31.depot 18% 136KB 25.2KB/s 0.0KB/s - stalled -

Output of ping:

[root@NCRCI]:/home/proy$ ping misci
PING misci.pptc.gc.ca: 64 byte packets
64 bytes from 10.2.156.46: icmp_seq=0. time=11. ms
64 bytes from 10.2.156.46: icmp_seq=1. time=19. ms
64 bytes from 10.2.156.46: icmp_seq=2. time=11. ms
64 bytes from 10.2.156.46: icmp_seq=3. time=11. ms
64 bytes from 10.2.156.46: icmp_seq=4. time=13. ms

----misci.pptc.gc.ca PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms) min/avg/max = 11/13/19

Output of traceroute:

[root@NCRCI]:/home/proy$ traceroute misci
traceroute to misci (10.2.156.46), 30 hops max, 40 byte packets
1 10.1.1.254 (10.1.1.254) 0.212 ms 0.250 ms 0.155 ms
2 10.1.1.250 (10.1.1.250) 2.344 ms 1.340 ms 1.137 ms
3 10.2.232.126 (10.2.232.126) 12.102 ms 12.562 ms 12.525 ms
4 10.1.30.254 (10.1.30.254) 11.420 ms 11.460 ms 11.135 ms
5 misci.pptc.gc.ca (10.2.156.46) 10.673 ms 10.859 ms 10.855 ms

All I've seen on the net regarding this has a dead end. No resolutions found. Some seem to be talking about NIC / Switch Port speeds mismatches... Checking them all :

[root@NCRCI]:/home/root$ lanadmin -x 0
Speed = 1000 Full-Duplex.
Autonegotiation = On.

[root@MISCI]:/home/root$ lanadmin -x 0
Speed = 1000 Full-Duplex.
Autonegotiation = On.

Both servers are set to auto_neg. Here's the /etc/rc.config.d/hpietherconf for them:

HP_IETHER_INTERFACE_NAME[2]=lan0
HP_IETHER_SPEED[2]=auto_on
HP_IETHER_AUTONEG[2]=On
HP_IETHER_MTU[2]=9000
HP_IETHER_STATION_ADDRESS[2]=0x001a4b08f934
HP_IETHER_FLOW_CONTROL[2]=On
HP_IETHER_SEND_CKO[2]=Off
HP_IETHER_RECV_CKO[2]=Off
HP_IETHER_VMTU[2]=0
HP_IETHER_SEND_MAX_BUFS[2]=1
HP_IETHER_SEND_COHP_IETHER_RECV_MAX_BUFS[2]=1
HP_IETHER_RECV_COAL_TICKS[2]=0
HP_IETHER_ITR_MODE[2]=-1
HP_IETHER_DIAG_THRESH[2]=0

Now if I try this from Server A to another HPUX v3 server on the SAME VLAN, it's fine! No problems!

So, this definitely seems to me like a network / switch problem! But the guys taking care of the switches say no. They've provided me with the following switch config outputs :

SERVER A:
interface GigabitEthernet0/17
description NCRCI NIC1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet0/18
description NCRCI NIC2
switchport access vlan 10
switchport mode access
shutdown
spanning-tree portfast
!


SERVER B:

interface GigabitEthernet1/0/25
description MISCI NIC2
switchport mode access
logging event trunk-status
no mdix auto
spanning-tree portfast
!
interface GigabitEthernet2/0/25
description MISCI NIC1
switchport mode access
logging event trunk-status
no mdix auto
spanning-tree portfast
!

Now I'm not a switch guy, so don't really know how to interpret these. But they seem to be saying Both are set to AUTO all the way. They seemed to be saying the GiGE works "better" with AUTO then forcing it to 1000FD. Now that got me confused cause if I remember in the past, it was always better to force speeds whenever possible... Whatever.

So, totally puzzled here. Can anyone spark a light on this one for me?

Thanks. Patrick

[Ganesan R]

Hi,

I strongly believe it is network part issue. Simple logic, If it is not working, then it should not work on the same VLAN too.

I have come across a issue once that, will not be able to ftp the files starting with 227*. Finally it was something to do with firewall/switch settings. Likewise network guys has to dig into their configurations and settings.

Unfortunately, i am also not a network guy to pin point where the issue lying.

Hopefully someone could point the exact issue here...

[PatRoy]

Thanks. But do take note that when I said it worked on the same VLAN, it was another HPUX 11.31 server. Let's name it ServerC. Same HW, same everything of course (incl. patches, etc). But it still worked! From A to C.

Also note, apparently (was not there when they tried) it's fine going from a Tru64 (same VLAN as ServerA) to ServerB... but I wasn't there and not yet sure of the switch configs (i.e. are they even going through the same switches, etc). I have to double check that.

[Rita C Workman]

The ports on those switches don't looke the same to me.

Server A
0/17 & 0/17 both shows switchport access vlan10
Server B
doesn't

Server B
Shows no mdix auto on 2/0/25 (I'm going to guess that mean auto negotiate)
1/0/25 doesnt
0/17 or 0/18 on Server A doesnt

I'm no networking person either, but they don't look like they are the same. Maybe they are....but to this laymen they look different.

I'd ask for confirmation on every port speed. And definitely a follow up on what Genesan mentioned.
Also remember that for ssh protocol transfer like sftp you need both 21 and 115 open. Most network guys have 115 shutdown.

Just a thought,
Rita

[PatRoy]

What I was told about the differences between the 2 switches:

logging event trunk-status: logs the trunk status changes if the interface was in "trunk" mode (which is NOT the case) as I am told.

no mdix auto: usually cisco switches have "uplink" ports which allows 2 switches to be connected together. If you're missing some of theses uplink ports, you can use any ports to achieve them by loading this into the port and using a crossover cable.

That was a straight conversion from french to english translation.. hope I didn't screw it up. Lol.

Still kinda fishy that he has that enabled on ServerB. But it just didn't seem to worry him at all.

[PatRoy]

Wait.. it says NO mdix auto... never mind. It's NOT configured, which is okay.

[Matti_Kurkela · Accepted solution]

Ping, traceroute and ssh usually send and receive very small packets. On the other hand, SFTP, scp and FTP will use as large packets as they can to optimize the transfer.

From the symptoms, it seems that something bad happens to large packets, but not to the small packets.

The question is, what causes it?

Normally, the systems can automatically determine the correct maximum packet size using a technique known as Path MTU Discovery. However, if an over-zealous firewall admin (or HP-UX sysadmin with an ipfilter software installed) has blocked the reception of all ICMP packets, Path MTU Discovery cannot work.

In that situation, if the MTU value configured for the HP-UX network interface is larger than the maximum packet size allowed at any point in the network route between the hosts, the excessively large packets will seem to just vanish (because the ICMP error message telling the host that it needs to use smaller packets has been blocked).

There may be other reasons too. I once encountered a switch that apparently had developed a failure in its packet buffer RAM chip. It did not block the packets, but any packet long enough to hit the faulty region was corrupted... which obviously caused the TCP protocol to discard it.

The ping command normally very short packets (64 bytes in length), but you can optionally make it send larger packets. HP-UX ping can also report other ICMP messages if you use the "-v" and "-p" options.

Try pinging with larger packets and see what happens:

ping -pv misci 1500

Adjust the packet size value (1500 in the example above) to find out what is the largest working packet size. Then ask the network guys if the length value you discovered rings any bells for them.

You might reduce the interface MTU to the largest working value as a workaround... but Path MTU Discovery is an essential part of the Internet these days. If it does not work, you are likely to have connectivity problems with other sites, too. (You cannot ask the whole world to configure reduced MTU values to communicate with you.)

MK

[PatRoy]

Hey MK.

Thanks for the very useful information! I managed to ping from NCRCI to MISCI with up to 1400 packet size. At 1500, it wouldn't work with :

[root@NCRCI]:/home/root$ ping -pv misci 1500
PING misci.pptc.gc.ca: 1500 byte packets
36 bytes from 10.1.1.254: icmp_type=5 (Redirect)
x00: x45000024
x04: xca900000
x08: xff01da05
x0c: x0a0101fe
x10: x0a01012f
x14: x0501613b
x18: x0a0101fa
x1c: x450005f0
x20: x24064000
x24: xfe01a1a6
x28: x0a01012f
x2c: x0a029c2e
x30: x08003941
x34: x4c870000
icmp_code=1
36 bytes from 10.1.1.254: icmp_type=3 (Dest Unreachable)
x00: x45000024
x04: xca910000
x08: xff01da04
x0c: x0a0101fe
x10: x0a01012f
x14: x03046957
x18: x000005dc
x1c: x450005f0
x20: x24064000
x24: xfe01a1a6
x28: x0a01012f
x2c: x0a029c2e
x30: x08003941
x34: x4c870000
icmp_code=4
new Path MTU = 1500

It then stalls. I'm still waiting on the network guys to get back to me on this one... Still trying to figure out if ICMP's are blocked or not between those 2 boxes. They're not fast to respond unfortunately...

[Florian Heigl (new acc)]

Hi,

have a short look at http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_example09186a008010edab.shtml

This might support the idea you have to change your mtu size, but - I think your issue is that they don't even allow you to send jumbo frames at ALL.

Otherwise you should run into your issues not before ping -s 8900 or so.

[Matti_Kurkela]

Apparently the size parameter of the HP-UX ping means "ICMP payload size": the total packet size is (payload size + IP header size), which in this case was 1520 bytes.

In the original message, you listed your HP_IETHER_MTU[2] having a value of 9000. That means your server is trying to use Jumbo Frames, but the second ICMP dump (Dest Unreachable, icmp_code=4 which means "Fragmentation Needed") you received indicates that:

1.) Path MTU Discovery works, at least between NCRCI and the gateway 10.1.1.254.

2.) 10.1.1.254 indicated that Jumbo Frames are *not* supported on this route. The ICMP packet indicates the limit is 1500, which is the plain old Ethernet MTU value.

I agree with Florian - you might try reducing your HP_IETHER_MTU value to the standard value of 1500 and see if that improves matters. (For the ping command, that means the largest working value for the size parameter in your network should be 1480.)

As even your original traceroute indicated, 10.1.1.254 does not gateway your connection to the 10.2.*.* network, but instead redirects you to 10.1.1.250. This is what causes the first ICMP dump ("Redirect") in the ping output.

Maybe the 10.1.1.250 gets confused by unexpected jumbo-frame packets instead of telling the sender to reduce the MTU?

MK

[PatRoy]

Turns out that internally, jumbo frames were supported, but not when going to MISCI, since this goes through a router NOT GigE.

Left the nics at 9000 MTU, but changed the default GW (10.1.1.254) to the standard 1500 MTU size. That way, we keep jumbo frames internally, but when leaving the building, it drops to 1500.

Thank you all!!

Pat