topic Re: ftp large files fail, small files succeed in Operating System - OpenVMS

ftp large files fail, small files succeed

Ron Kaledas — Tue, 13 Jul 2010 18:08:59 GMT

Okay, I've searched for this, found a few matches, but nothing newer than 2006...(!) Those fixes didn't help/apply to my issue.

I'm into a client via vpn, though I don't think that should matter. (though once I get connected via vpn, I can't putty directly to the test system, I have to putty to the prod system and then telnet to the test system.)

Anyway, trying to ftp a large file from prod to test (both vms). I can ftp smaller files. for larger ones, I get the output below. it connects enough to get the filesize, and allocates that space on the test system disk. but, the file size remains 0.

Does anyone still hang out here? :) any thoughts? some type of firewall/network switch setup issue that I'd have no access to? (I did ask them to verify fixed full on the ports.)

Thanks!
Ron

FTP> bin
200 TYPE set to IMAGE.
FTP> get openvms_alpha_8_3.zip
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSMGR.WEB.83]openvms_alpha_8_3.zip; (10.252.16.75,49183) (209548312 bytes)
HNATST::_TNA8: 14:39:12 TCPIP$FTP CPU=00:00:00.72 PF=1386 IO=2374 MEM=349
GET (VMS+) 0 bytes 00:00:04.98 elapsed (0.00 KB/S)
Local: LAB2:[000000]OPENVMS_ALPHA_8_3.ZIP;1
Remote: openvms_alpha_8_3.zip
HNATST::_TNA8: 14:39:22 TCPIP$FTP CPU=00:00:00.72 PF=1386 IO=2378 MEM=349
GET (VMS+) 0 bytes 00:00:15.14 elapsed (0.00 KB/S)
Local: LAB2:[000000]OPENVMS_ALPHA_8_3.ZIP;1
Remote: openvms_alpha_8_3.zip
HNATST::_TNA8: 14:45:59 TCPIP$FTP CPU=00:00:00.72 PF=1386 IO=2382 MEM=349
GET (VMS+) 0 bytes 00:06:51.61 elapsed (0.00 KB/S)
Local: LAB2:[000000]OPENVMS_ALPHA_8_3.ZIP;1
Remote: openvms_alpha_8_3.zip
%SYSTEM-F-CONNECFAIL, connect to network object timed-out or failed

Re: ftp large files fail, small files succeed

Andy Bustamante — Tue, 13 Jul 2010 19:14:22 GMT

Ron,

I'd start with looking at the network. Make sure both VMS systems and switches are set to agree on speed/duplex or auto negotiate.

What sort of VPN and is there an MTU recommendation to trim MTU? Are these Giga adaptors? If so, are jumbo frames enabled and over running the VPN?

Andy

Re: ftp large files fail, small files succeed

Robert Gezelter — Tue, 13 Jul 2010 20:33:48 GMT

Ron,

As has been noted, duplex mismatches somewhere in the path will produce this type of symptom (I have seen it, they can be pernicious to track down -- I had it happen at a client a while back, someone in the network group had swapped a switch and mis-configured the replacement).

A LAN trace of the transfer can be illuminating. Wireshark is freely available, and runs on many standard personal platforms.

The small files may not hit the timing problem that is the actual problem.

- Bob Gezelter, http://www.rlgsc.com

Re: ftp large files fail, small files succeed

John Gillings — Tue, 13 Jul 2010 21:02:40 GMT

Ron,

As others said, this is almost certainly a duplex mismatch.

> (I did ask them to verify fixed full on the ports.)

Please ask "them" to set AUTO on all ports and switches. Hard setting network speeds is just setting yourself up for a failure like the one you're observing sometime in the future.

Remember to check all systems in the path between you and the FTP target.

Re: ftp large files fail, small files succeed

Robert Brooks_1 — Tue, 13 Jul 2010 23:45:27 GMT

The frequent response to these problems by the
VMS Engineer who wrote ethernet drivers for a couple of decades is to set both ends of the connection to autonegotiate, assuming a modern version of VMS (in this case "modern" means V.3-2 and newer) and a properly-functioning switch. VMS should always get it right.

-- Rob

Re: ftp large files fail, small files succeed

Ron Kaledas — Wed, 14 Jul 2010 10:55:57 GMT

Well, everyone (and hi to John and Andy!) seems to agree that it is a network configuration problem, so I will pursue that further. I haven't actually heard back from "them" yet.

I do find it interesting though, I'd always heard NOT to use auto, but that seems to be the opposite of what I'm hearing here. I will take that into consideration! It seems to me - though I can't remember details - that auto was to be avoided because "things" (cisco, nics, can't remember what) didn't always make the right choice, so that's why fixed was preferred. Of course, this may be outdated information...

So, thanks for your input, and I will report back if I find a definitive answer.

Re: ftp large files fail, small files succeed

Richard Whalen — Wed, 14 Jul 2010 11:30:37 GMT

Though I won't say don't investigate the network configuration, I think that there may be other things to consider as well.

Since small file transfer fine, but large files are allocated, but not transferred, I'd look for something that might be tearing down an "idle" connection. Does the path between the production system and test system involve some sort of firewall or NAT device?

The information above shows that you aren't operating in passive mode; have you tried passive mode?

Re: ftp large files fail, small files succeed

Ron Kaledas — Wed, 14 Jul 2010 11:35:08 GMT

Guess I'm not sure (/don't remember) what passive mode is, could clarify how to use it?

Re: ftp large files fail, small files succeed

Richard Whalen — Wed, 14 Jul 2010 13:24:34 GMT

To enter passive mode, use the SET PASSIVE ON command before starting the transfer.
This will cause the FTP server to create the data port and pass information to the client as to how to connect to it.

Re: ftp large files fail, small files succeed

Hoff — Wed, 14 Jul 2010 13:32:21 GMT

Note: ACTIVE and PASSIVE are likely not involved here.

ACTIVE and PASSIVE are transfer modes. Check your ftp client command documentation for details. The command is often a toggle command "passive", but syntax varies. TCP/IP Services uses the SET PASSIVE (on|off) command.

ACTIVE: the ftp client tells the ftp server which port the server should connect back to. Can be blocked by the client or client network firewall.

PASSIVE: client asks server for the identity of a port to connect into. Can be blocked by the server or server network firewall.

Because of that second channel and its associated handling, the ftp protocol design is largely incompatible with modern IP networks; with (most) IP firewalls.

(Trivia: ftp is older than IP, and way older than IP firewalls.)

The "fun" is that ftp uses a second IP port from the ephemeral range, and in a way that is inherently blocked by the most prevalent designs for IP firewalls. This means that the port range specified for ftp will have to be coordinated with the firewall, and opened. (And yes, other network applications can also use the ephemeral port range, so various firewall administrators are loathe to open it.)

There are comparatively sophisticated IP firewalls that can deal with ftp, as they explicitly know the ftp protocol, sniff and remember the ftp traffic, and automatically open the correct port for the impending data connection. These firewalls aren't yet in common use.

All of which means that ftp goes off the rails fairly regularly, and folks try the so-called active and passive transfers, and can end up opening up vast ranges of IP ports.

ftp also spews your credentials in cleartext, so it's a poor choice for any applications where write access is required.

sftp is a far newer design and - though it shares three letters with and its basic purpose with ftp - shares little else. sftp is vastly easier to punch through a firewall, and you can also incorporate certificates to greatly reduce your exposure to brute-force server attacks.

Yes, I've been known to express a relative distaste for ftp. Here are some technical details behind that opinion and around why ftp is such utter "fun" to deal with:

http://labs.hoffmanlabs.com/node/530

ftp is best left to the task of anonymous reading of and copying of files from a server, and little else. If you even use that and not something like WebDAV.

A network that requires VPN to PuTTY to telnet (and particularly if that's three separate steps and two intermediate hops) would imply the potential for simplification there, and potentially elsewhere in the network design. That is not a typical sequence for a secure network, and not an access sequence that would be commonplace for a more "open" internal network. I'd probably install a better and VPN-capable firewall, or otherwise reconfigure the firewall(s) and that LAN to allow a connection onto the LAN with the VMS servers. Based in this description (and I've built and rebuilt these sorts of network configurations myself) the network design looks to need some assistance to better contend with the organic growth it has apparently undergone.

Again, Note: IP firewalls aren't a factor with VMS, as VMS doesn't (yet?) have one. This particular case likely does NOT involve firewalls, particularly given the transfer works with smaller files.

Re: ftp large files fail, small files succeed

Ron Kaledas — Wed, 14 Jul 2010 13:46:29 GMT

Thanks for all the input.

I have zero control or input on the network side of things at this site, I can only (try to) use what they give me. Based on what I've read here, I'll see if I can get the ports that my systems connect to set to auto-negotiate if I can. Does that apply to both duplex and speed, or are there separate auto settings for each? (I'm not a network guru!) mc lancp tells me I'm expecting 100 full on both. I can't take the prod system down to verify console settings.

fwiw, both systems are at vms 7.3-2. (upgrade is in progress to 8.3, wish I could go right to 8.4, but can't.)

Re: ftp large files fail, small files succeed

Mike Maltsman — Wed, 14 Jul 2010 13:51:19 GMT

It is possible that FTP server closes COMMAND port 21 before closing high data port. You may try to increase timeout interval. This has it's own issues with having too many ports open in TIME_WAIT. What is interposing that you can actually still be able to transfer the file successfully even with that error.

Re: ftp large files fail, small files succeed

Jon Pinkley — Wed, 14 Jul 2010 15:41:21 GMT

Ron,

Without more information than you have provided, all we can do is make some educated guesses.

Has ftp ever worked between these systems? If so, what has changed?

Does ftp work between other systems and the production system?

Does ftp work between other systems and the test system?

What do you mean by "I can ftp smaller files."? 1 block, 10 block, 100 blocks, 1000 blocks? Can you show us an example of what works?

If a small file works, I don't see how the problem could be related to passive mode. That only affects which side opens the data connection, and is related to firewall configurations. Using put from the other system could also test this.

We need to know how the test system is connected to the production system. Is there something more than an ethernet switch (layer 2) between the two systems; a router, firewall, NAT, VPN etc.?

A layer 2 ethernet switch isn't going to modify packets, there could be duplex mismatches, but these should be visible in the output from LANCP> show device /internal_counters.

Routers and gateways provide for many more possible failures.

PMTUD blackhole

https://supportforums.cisco.com/docs/DOC-5839;jsessionid=55857038EDD78889298385026D39F4E2.node0

Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC (this has a lot of good technical info that applied to VPN connections, but this doesn't explain how to avoid the problems with VMS)

http://www.cisco.com/application/pdf/paws/25885/pmtud_ipfrag.pdf

If there is more than just a simple switch between the two VMS systems, you could try this to see if it makes a difference. If it doesn't, then return to previous state. You may need to apply to both sides.

The following can possibly work around a PMTUD blackhole, but it could also increase the TCP/IP overhead (since it can cause smaller packets to be sent)

Contents of SYS$COMMON:[SYSMGR]TCPIP_SYSCONFIG.COM

$ @sys$manager:tcpip$define_commands.com
$ sysconfig -r inet tcp_mssdflt=536
$ sysconfig -r inet pmtu_enabled=0
$ exit

Good luck,

Jon

Re: ftp large files fail, small files succeed

Hoff — Wed, 14 Jul 2010 17:32:14 GMT

Call up Mike LeRoy. Explain the problem. Done.

Re: ftp large files fail, small files succeed

Ron Kaledas — Wed, 14 Jul 2010 18:20:24 GMT

Nice thought Steve...however, this isn't at dmc, it's at another site. And, it so happens that I was on a con call with that site today, which included the CIO there, and I mentioned the problem...

it's being worked on. I don't expect much this week...

Re: ftp large files fail, small files succeed

Ron Kaledas — Wed, 14 Jul 2010 18:22:47 GMT

As for Jon's questions - this is a new test system, that got set up for this upgrade. Also, I only have these 2 vms systems to go between.

Sorry, I don't remember the rest of your questions, but that should answer a few of them. (wish I could see the thread when I'm doing a reply!)

Re: ftp large files fail, small files succeed

Jon Pinkley — Wed, 14 Jul 2010 19:41:56 GMT

Your original post indicates that no data was transferred in the data connection, so that makes me think it is could be related to an MTU black hole. It is possible that short packets get through, but large ones are being dropped by something in the path between the systems. If there is something blocking ICMP packets, MTU path discovery will not work.

Q1: What is the largest small file that you can successfully ftp?

Q2: Does $ ping -s 1500 10.252.18.75 work? If not what about smaller values for size, for example -s 1200? Use binary search to determine the largest packet you can get to work.

To be able to see questions while answering, cut and paste to notepad, then use notepad to edit your response, then select your response from notepad, and paste into ITRC.

Jon

Re: ftp large files fail, small files succeed

Andy Bustamante — Wed, 14 Jul 2010 20:27:56 GMT

Ron,

You may also try traceroute with the -f flag

from a 7.3-2 system

$tcpip traceroute -f 10.92.212.57 1500
1 GATEWAY (10.40.92.1) 0.976 ms 0.976 ms 0.0 ms
2 172.20.252.6 (172.20.252.6) 0.977 ms 0.977 ms 0.976 ms
3 10.198.10.19 (10.198.10.19) 0.976 ms 0.976 ms 0.976 ms
4 10.198.10.19 (10.198.10.19) 0.977 ms (ttl=253!) !F=1412 0.976 ms (ttl=253!) !F=1412 0.977 ms (ttl=253!) !F=1412
(switching to new packet size 1412)
5 DRES40 (10.198.212.57) 1.95 ms 1.95 ms 1.95 ms

the -f is don't fragement, 1500 is packet size. There is an encrypted tunnel in the example above.

I didn't expect TCPIP to give a working MTU in one step. Kudos to the TCPIP team.

Re: ftp large files fail, small files succeed

John Gillings — Wed, 14 Jul 2010 23:36:50 GMT

Ron,

>It seems to me - though I can't remember
>details - that auto was to be avoided
>because "things" (cisco, nics, can't
>remember what) didn't always make the right
>choice, so that's why fixed was preferred.
>Of course, this may be outdated
>information...

Very outdated! It might have been relevant last century with a limited set of very old adapters, but for at least the last 10 years, autonegotiate has worked correctly on all OpenVMS systems.

Bad things happen when one end of the link is hard set and the other end is set to auto (and since auto is an all but universal default, this happens a lot if you hard set ports anywhere). The autonegotiate end typically sets the speed correctly, but chooses half duplex. That works intermittently if the other end is set to full duplex, typically failing under load. Blame the poorly thought out autonegotiate protocol dreamt up by those who claim to be the setters of industry standards, but suffice to say that if everyone agrees to autonegotiate it will work reliably.

From a support perspective, I have a policy of checking ethernet port settings for ANY performance related issue on an OpenVMS host. I never cease to be amazed at how pervasive this configuration error could be! Anything from FTP to cluster locking and sometimes things that you'd never guess could be affected by network behaviour.

My strong recommendation is for ALL ports on ALL systems and network infrastructure for ALL operating systems to be set to autonegotiate (note that for gigabit ethernet it's mandatory - I don't know why anyone even allows for it to be turned off!).

The only good thing to say about hard setting port speeds is it keeps a lot of folk like me employed, fixing them!

Re: ftp large files fail, small files succeed

Steve Reece_3 — Fri, 16 Jul 2010 17:46:33 GMT

Stupid though it may sound, if you're stuck with FTP rather than sftp, try putting the command
ftp> hash
in before you start the transfer. It will at least tell you if there are any data going over the link. Others will confirm whether it keeps the control link alive too.

It will create lots of output though on the terminal/log file doing the transfer!

Steve