topic Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files) in Operating System - OpenVMS

%TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Mon, 24 Oct 2005 16:30:42 GMT

After installing OpenVMS 7.3-2 and TCPIP V5.4 - ECO 5, putting large files via FTP from an Alpha to a IBM system does not work anymore. The remote (IBM) system seems to see the session as idle and disconnects it after 5 minutes.

The file to put is about 10,5 Mb and gets the next error:

$ ftp IBMSYSTEM
220-GIDSFTP1 IBM FTP CS V1R4 at BG08, 18:22:47 on 2005-10-24.
220 Connection will close if idle for more than 5 minutes.
Connected to IBMSYSTEM.
Name (IBMSYSTEM:cnrennes): username
331 Send password please.
Password:
230 USERNAME is logged on. Working directory is "USERNAME.".
FTP> put INTERTMP:CL_250.NIEUW_AP RI.DATA.XXX.FTP
200 Port request OK.
125 Storing data set RI.DATA.XXX.FTP
()
STVMS1::CNRENNES_1 18:22:24 TCPIP$FTP CPU=00:00:44.73 PF=37668 IO=189408 MEM=327
PUT (ASCII) 61464 bytes 00:00:02.23 elapsed (26.83 KB/S)
Local: DISK99:[SINX_INTER_DAT.TMP]CL_250_24_B_M_IC_RUN_XCAS.NIEUW_AP;1
Remote: RI.DATA.ST25024X.FTP

%TCPIP-E-FTP_SEND, cannot send on socket
-SYSTEM-F-LINKDISCON, network partner disconnected logical link
525 No data is available on the data connection
FTP> ()
STVMS1::CNRENNES_1 18:27:24 TCPIP$FTP CPU=00:00:44.73 PF=37711 IO=189421 MEM=327
FTP> Exit

Of course before the upgrade (with OpenVMS 7.1-2 and UCX V4.2 - ECO 5) there was no problem putting large files to the IBM system.
And on the site of the IBM system there were no changes.

Putting a small file to the IBM system is still working all right.

Who can help me with this!

Regards,
Arno van Rennes

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Karl Rohwedder — Mon, 24 Oct 2005 16:45:26 GMT

Arno,

was anything else changed? The speed of 26KB/sec sounds quite slow to me (or is it, because it's an IBM :-)?

regards Kalle

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Mon, 24 Oct 2005 17:00:39 GMT

Hi Kalle,

No, as far as I know, nothing else has been changed.
Sending verry large files (>200000 blocks) between our own VMS systems runs at 5917KB/sec.
So that must be the IBM-system :-).
(or our setting?)

Regards,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Andy Bustamante — Mon, 24 Oct 2005 17:39:05 GMT

Is there a firewall between the IBM and VMS systems? See the TCPIP release notes section 3.9

http://h71000.www7.hp.com/doc/732FINAL/TCP_RN/tcp_rnpro_002.html#ftp_problems

Previous versions of TCPIP would accpet ftp-data on something other than port 20. This showed a few firewall issues. You can restore the old behavior or check the firewall configuration.

Andy

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Michael Yu_3 — Mon, 24 Oct 2005 19:23:40 GMT

Hi Arno,

The problem was probably caused by the firewall which kills TCP sessions after they have been idle for 5 minutes.

FTP sent data on the data channel but the control channel will be idle. I think the firewall has terminated the control channel and hence the data channel is also disconnected.

UCX V4.2 worked because the keepalive timer was set to 75 seconds. TCP/IP V5.4 is different. By default, keepalive is disabled (tcp_keepalive_default = 0) and even if it is enabled, the default keepalive timer is 7200 seconds (tcp_keepidle = 14400).

If my guess is correct, this problem can be solved by enabling keepalive and by setting the keepalive timer shorter than 5 minutes (300 seconds).

So before you FTP to the IBM, try the following.

$ @sys$manager:tcppip$define_commands
$ sysconfig -r inet tcp_keepalive_default=1
$ sysconfig -r inet tcp_keepidle=500

The above will enable keepalive and set the keepalive timer to 500 half-seconds, i.e. 250 seconds.

Hope the above helps.

Remember the changes above will affect all new TCP connections, you may choose to restore the original settings after the FTP. Also these settings are volatile, they will not survive reboots unless you make them permanent using sysconfigdb.

Thanks and regards.

Michael

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Tue, 25 Oct 2005 00:46:32 GMT

Hi Michael,

Unfortunately, setting tcp_keepalive_default=1 and tcp_keepidle=500 makes no difference.

After exactly 5 minutes:
%TCPIP-E-FTP_SEND, cannot send on socket
-SYSTEM-F-LINKDISCON, network partner disconnected logical link
525 No data is available on the data connection

Thanks,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Tue, 25 Oct 2005 00:59:58 GMT

Hi Andy,

I read the release notes but I don't see a direct relation whit this issue.
The way I read it there will be no connection at all, and in this case small files can be put to the IBM system, only the connection with large files is terminated after 5 minutes.
Please correct me if I'm wrong.

Thanks,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Wim Van den Wyngaert — Tue, 25 Oct 2005 03:38:34 GMT

Arno,

I would say that the FTP server on the IBM aborted and VMS stayed alive and the connection was stopped by keepalive. Check on the IBM. How much of the file arrived ?

If there is a firewall, I once had a problem with it. Read the enclosure. It talks about a trace I sent them of an analogue problem.

Wim

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Tue, 25 Oct 2005 07:47:32 GMT

Hi Wim,

I cannot check on the IBM system, this is at a bank.
I did talk to then and they don't see anything faulty when I send the large file.

I have tested it with the settings in your attachment:
ipport_userreserved_min = 1024
ipport_userreserved = 5000
tcp_keepalive_default = 1
tcp_keepidle = 500

and

ipport_userreserved_min = 1024
ipport_userreserved = 5000
tcp_keepalive_default = 0
tcp_keepidle = 14400

Unfortunately this also did not work.

I've also set the passif mode from "Passive is AUTO (IPv4: OFF, IPv6: ON)" to "Passive is OFF" but also no results.

Thanks,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Wim Van den Wyngaert — Tue, 25 Oct 2005 08:57:29 GMT

Arno,

I also work in a bank and know how you feel.

Did the IBM receive part of the file ?

Wim

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Tue, 25 Oct 2005 09:15:15 GMT

Hi Wim,

If I disconnect the session while putting the file, I see it on the FTp directory on the IBM system as:

Volume Unit Referred Ext Used Recfm Lrecl BlkSz Dsorg Dsname
PRO894 3390 **NONE** 1 37 VB 4100 27998 PS RI.DATA.ST25024X.FTP

After a few minutes it disappears.

Regards,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

rick jones — Tue, 25 Oct 2005 11:47:55 GMT

The FTP idle timeout is likely unrelated to the TCP level keepalives. TCP keepalives will not progress beyond the TCP layer, so FTP above it will still think the FTP control connection is "idle."

So, if it is indeed the case that the IBM system is upset that it hasn't heard anything on the control connection for five minutes, that means one either has to figure-out why the data transfer is taking longer than five minutes, or convince the IBM system to have a longer idle control timeout.

I'm not a VMS guy (just plain networking :) but on the FTP clients I've used, one can use the "idle" command to request a different idle timeout on the control connection. The FTP server retains the last word, but it might be worth a shot.

As for the speed of transfer, might be interesting to check the TCP statistics on the sending side, and receiving side if you can. Look for things like retransmissions and retransmission timeouts. I've no idea how much work it would take to compile under VMS, but "beforeafter" from ftp.cup.hp.com dist/networking/tools/ can be used to subtract one file of network statistics from another (at least the netstat format one finds under HP-UX and linux)

If you can check link-level stats that would be good as well. While transfers to the local VMS systems suggest that the link-level is OK, this boilerplate on duplex may be of interest:

$ cat usenet_replies/duplex
How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

Auto Half Full

Auto Happiness Lucky Sorrow

Half Lucky Happiness Sorrow

Full Sorrow Sorrow Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly :) Sorrow means that there _will_ be a duplex
mis-match.

When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of late collisions. On
the side running full-duplex you will see things like FCS errors.
Note that those errors are not necessarily conclusive, they are simply
indicators.

Also, if the IBM system is "far away" - in terms of latency - eg ping times if there was any change in default TCP window/socket buffer sizes with the VMS change that may have affected the transfer speed. One limit to TCP transfer speed is:

WindowSize/RoundTripTime

While the IBM system is likely advertising the same window size as before, perhaps the VMS system is no longer taking full advantage - "WindowSize" here would be the smaller of the receiver's window, the sender's equivalen to the SO_SNDBUF, and the sender's congestion window.

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Stanley F Quayle — Tue, 25 Oct 2005 14:31:08 GMT

> I've also set the passif mode from "Passive
> is AUTO (IPv4: OFF, IPv6: ON)" to "Passive is
> OFF" but also no results.

You want to set Passive to ON. This forces the transfer to use the control channel, which makes the channel appear to be active at all times.

This assumes, of course, that the remote end supports Passive...

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

rick jones — Tue, 25 Oct 2005 15:12:41 GMT

Stanley - Unless I'm mixing passives, passive in FTP does _not_ use the control channel. It simply reverses the roles of passive acceptor and active connector on the data connection.

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Thu, 27 Oct 2005 09:28:20 GMT

Hi Rick,

The window size could be an issue here!
With tcpdump, I see some action, at the start of the ftp-put session:
The window size on both sides is 65535 but the IBMSYSTEM keeps changing the window size.

See:
14:08:42.633574 STVMS1.3834 > IBMSYSTEM.21: S 1365984768:1365984768(0) win 65535
14:08:47.668408 STVMS1.3834 > IBMSYSTEM.21: S 1365984768:1365984768(0) win 65535
14:08:47.712351 IBMSYSTEM.21 > STVMS1.3834: S 2390338092:2390338092(0) ack 1365984769 win 65535
14:08:47.712351 STVMS1.3834 > IBMSYSTEM.21: . ack 1 win 65535
14:08:47.786565 IBMSYSTEM.21 > STVMS1.3834: P 1:64(63) ack 1 win 65535
14:08:47.820742 STVMS1.3834 > IBMSYSTEM.21: . ack 64 win 65535
14:08:47.868591 IBMSYSTEM.21 > STVMS1.3834: P 64:124(60) ack 1 win 65535
14:08:47.881285 STVMS1.3834 > IBMSYSTEM.21: P 1:15(14) ack 124 win 65535
14:08:48.022878 IBMSYSTEM.21 > STVMS1.3834: P 124:151(27) ack 15 win 65528
14:08:48.023854 STVMS1.3834 > IBMSYSTEM.21: P 15:30(15) ack 151 win 65535
14:08:48.196695 IBMSYSTEM.21 > STVMS1.3834: P 151:212(61) ack 30 win 65527
14:08:48.196695 STVMS1.3834 > IBMSYSTEM.21: P 30:42(12) ack 212 win 65535
14:08:48.281650 IBMSYSTEM.21 > STVMS1.3834: P 212:278(66) ack 42 win 65529
14:08:48.421290 STVMS1.3834 > IBMSYSTEM.21: . ack 278 win 65535
14:08:59.582958 STVMS1.3834 > IBMSYSTEM.21: P 42:67(25) ack 278 win 65535
14:08:59.630807 IBMSYSTEM.21 > STVMS1.3834: P 278:300(22) ack 67 win 65522
14:08:59.631783 STVMS1.3834 > IBMSYSTEM.21: . ack 300 win 65535
14:08:59.631783 STVMS1.3834 > IBMSYSTEM.21: P 67:93(26) ack 300 win 65535
14:08:59.778258 IBMSYSTEM.20 > STVMS1.4678: S 2390581620:2390581620(0) win 32768 14:08:59.778258 STVMS1.4678 > IBMSYSTEM.20: S 1182583056:1182583056(0) ack 2390581621 win 65535
14:08:59.819271 IBMSYSTEM.20 > STVMS1.4678: . ack 1 win 32768
14:08:59.900321 IBMSYSTEM.21 > STVMS1.3834: P 300:350(50) ack 93 win 65522
14:08:59.909109 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:08:59.909109 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:08:59.909109 STVMS1.4678 > IBMSYSTEM.20: . 2921:4381(1460) ack 1 win 65535 (DF)
14:09:00.032148 STVMS1.3834 > IBMSYSTEM.21: . ack 350 win 65535
14:09:00.717651 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:09:00.717651 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:09:03.228233 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:09:03.228233 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:09:08.739599 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:09:08.739599 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:09:20.278196 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:09:20.278196 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:09:43.842391 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:09:43.842391 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:10:31.514965 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:10:31.515942 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:11:35.224599 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:11:35.224599 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:12:38.938138 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:12:38.938138 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:13:42.651678 STVMS1.4678 > IBMSYSTEM.20: . 1:1461(1460) ack 1 win 65535 (DF)
14:13:42.651678 STVMS1.4678 > IBMSYSTEM.20: . 1461:2921(1460) ack 1 win 65535 (DF)
14:13:59.889129 IBMSYSTEM.20 > STVMS1.4678: R 2390581621:2390581621(0) win 32768
14:13:59.906706 IBMSYSTEM.21 > STVMS1.3834: P 350:399(49) ack 93 win 65522
14:13:59.915495 STVMS1.3834 > IBMSYSTEM.21: . ack 399 win 65535

There is a clear difference in the wscale between both sides, wscale 0on my side and wscale 1 on the other.

Does one of the VMS cracks know how to change the wscale?
I have tried both:
$ tcpip set protocol tcp /window_scale
and
$ sysconfig -r inet tcp_dont_winscale=0
to get the Window scale: enabled

But with a new tcpdump the wscale is still 0.

Thanks,
Arno

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

rick jones — Thu, 27 Oct 2005 11:21:42 GMT

Well, that the IBM system is not always showing 65535 in the window field of the ACK's it is sending is "normal" - it ise sending ACKs and saying how much space (remember to multiply by two since the scale factor is one) in the window remains. When you see the window field getting smaller like that, it means that data is arriving, but not being consumed by the remote application.

The second half of your trace is more disturbing - it is showing a boatload of TCP retransmissions and no ACK's from the IBM system. That would do some really nasty things to performance and I think I saw a ReSeT segment in there as if one side gave up (IIRC it was the VMS side, probably after retransmit timing-out).

Re: %TCPIP-E-FTP_SEND, cannot send on socket (for large files)

Arno van Rennes — Fri, 04 Nov 2005 06:15:59 GMT

Well I have found the solution!

- This is what I have changed (and did not help):

socket:
somaxconn = 1024 to 65535 (recommended by HP)
sominconn = 1 to 65535 (recommended by HP)

inet:
ipport_userreserved = 5000 to 65535 (recommended by HP)
ipport_userreserved_min = 1024 to 49152 (recommended by HP)
tcp_keepidle = 250 to 14400 (default value)
tcp_keepalive_default = 0 to 1 (and back to 0 again)
tcp_recvspace=61440 to 65535 (and back to 61440 again)
tcp_sendspace = 61440 to 65535 (and back to 61440 again)
tcp_ttl = 60 to 128 (and back to 60 again)

-----------------------------------------

* And this is what did the trick:

$ sysconfig -r inet pmtu_enabled=0
(pmtu_enabled was enabled)

HP indicates:
(in http://h71000.www7.hp.com/doc/732final/documentation/pdf/aa-rn1vb-te.pdf)
2.1.6.10 Disabling Use of the PMTU Protocol:
Packets transmitted between servers are fragmented into units of a specific size in order to ease transmission of the data over routers and small-packet networks, such as Ethernet networks. When the inet subsystem attribute pmtu_enabled is enabled set to 1, which is the default behavior), the system determines the largest common path maximum transmission unit (PMTU) value between servers and uses it as the unit size. The system also creates a routing table entry for each client network that attempts to connect to the server.
Performance Benefits and Tradeoffs:
If a server handles traffic among many remote clients, disabling the use of a PMTU can decrease the size of the kernel routing table, which improves server efficiency. However, on a server that handles local traffic and some remote traffic, disabling the use of a PMTU can degrade bandwidth.
When to Tune:
If an Internet server has poor performance and the routing table increases to more than 1000 entries, you should disable the use of PMTU. This is also recommended if you have a server that handles traffic among many remote clients.
Recommended Values:
To disable the use of PMTU protocol, set the value of the pmtu_enabled attribute to 0.

All other FTP sessions keep function the same way as before so I presume that I can leave it this way!

Thanks to all for your input!

Regards,
Arno