topic Re: COPY/FTP error in Operating System - OpenVMS

COPY/FTP error

gunners — Thu, 30 Jun 2011 10:21:45 GMT

Hi there folks, I am getting the error below when trying to copy/ftp

Decnet copy works fine but someone suggested that I use copy/ftp as some of the files are huge and were taking hours to copy across. (the password below I have as 'password' for obvious reasons)

copy/ftp/lo DSA1501:[ibid91_sync_savesets]dsa1200.bak ibid91"system password"::dsa1501:[temp_backups]*.*

IBID90::_FTA298: 10:30:44 TCPIP$FTP CPU=00:07:45.14 PF=19148 IO=8804975 MEM=534

IBID90::_FTA298: 10:30:45 TCPIP$FTP CPU=00:07:45.15 PF=19148 IO=8804976 MEM=534

IBID90::_FTA298: 10:30:45 TCPIP$FTP CPU=00:07:45.15 PF=19148 IO=8804977 MEM=534

%TCPIP-E-FTP_NETERR, I/O error on network device

-SYSTEM-F-TIMEOUT, device timeout

Re: COPY/FTP error

Ian Miller. — Thu, 30 Jun 2011 10:59:33 GMT

why where the copies using DECnet taking a long time?

suggests all is not well with your network.

Re: COPY/FTP error

gunners — Thu, 30 Jun 2011 12:05:24 GMT

Hi There Ian,

I should have mentioned that the servers are in two locations - about 10 miles apart. The network is pretty good by all accounts and even with the major traffic from me yesterday (about 8million blocks) worth of files , there was no impact to users of other systems (who also use this link)

Re: COPY/FTP error

Volker Halle — Thu, 30 Jun 2011 15:10:20 GMT

Hi,

you've typed CTRL-T a couple of times, but the IO counter only increased by 1 each time, so there were no IOs being done by COPY/FTP. And the copy finally failed, so there has been a network connectivity issue.

If there are packets getting lost on that link between the 2 systems, DECnet will take a lot time to retransmit packets, which will make the perceived 'performance' look bad.

Volker.

Re: COPY/FTP error

John Gillings — Thu, 30 Jun 2011 21:20:56 GMT

gunners,

This sounds like a duplex mismatch to me. Check that the entire path between systems either has matching settings between NIC and switch ports. I recommend AUTONEGOTIATE everywhere, but if that's not possible, ensure that if any port has a hard setting, that it matches exactly at both ends.

If one end is st to (say) 100 Full Duplex and the other end is AUTO, the auto end will configure as 100 Half Duplex. On a lightly loaded network this is rarely an issue. So things like telnet "work fine". However, when you start loading it with things like file transfers the half duplex end will start missing packets and requesting retransmissions, leading to excessive transfer times and/or timeouts.

Many years ago a myth developed that OpenVMS doesn't handle network autonegotiation. It may have been true of mixtures of specific nics, brands of switches and (ancient) versions of network software, but for anything produced in the last decade or two, everything just works. If your site policies have a voodoo belief in "autonegotiate bad", please try to change it. (On the other hand, the continued prevalence of duplex mismatches keep many support staff in work, with the ability to easily achieve apparently miraculous fixes)

Re: COPY/FTP error

gunners — Tue, 05 Jul 2011 05:50:28 GMT

John , that sounds like a strong possibility. How would I check the settings ? - something like SHOW DEVICE 'device name' / FULL ?

Re: COPY/FTP error

Volker Halle — Tue, 05 Jul 2011 06:12:33 GMT

Does PING or TELNET work between those 2 systems ?

You can look at the console messages issued by the LAN driver with $ MC LANCP SHOW DEVICE/INTERNAL

This will show you 'possible duplex mismatch' errors detected and reported by the LAN driver. It will also show you the LAN startup messages, which should tell you the settings of the LAN device(s) at console level, and the link state.

Then you need to talk to your network people about the status of the switch ports, to which the LAN interfaces are connected.

Volker.

Re: COPY/FTP error

gunners — Tue, 05 Jul 2011 14:33:56 GMT

Hi there Volker ,

Yes indeed telnet works fine. I ran that LAN command on the 'receiving' server.

It seems from what I can make out that it has correct settings from some of the output here ?

000002A4 Requested link state <FlowControl, Fdx, 1000 mb,

Auto-negotiation>

000002A5 Current link state <FlowControl, Fdx, 1000 mb,

Auto-negotiation, Link up>

1 Link up timer

Re: COPY/FTP error

Volker Halle — Tue, 05 Jul 2011 15:58:56 GMT

That looks fine.

And how about the sending server ?

Volker.

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 07:55:52 GMT

Hi again Volker ,

ok here is a snapshot from the 'sending' servers config using the same LAN command

9856 Jumbo buffer size (bytes)

9018 Jumbo packet size (bytes)

000002A4 Requested link state <FlowControl, Fdx, 1000 mb,

Auto-negotiation>

000002A5 Current link state <FlowControl, Fdx, 1000 mb,

Auto-negotiation, Link up>

00001010 Driver flags <82571EB (Ophir), Copper>

00000004 Driver state <RunUp>

PCI-Express BUS type

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 07:58:15 GMT

sorry i meant sending 'server' as in singular

Re: COPY/FTP error

Volker Halle — Wed, 06 Jul 2011 08:07:44 GMT

So the sending and receiving servers do not seem to have a problem with autonegotiation settings, both are set to AUTO. And I assume, that there were no 'possible duplex mismatch' messages from the LAN drivers as shown at the bottom of the LANCP SHOW DEV/INT output.

But how about JUMBO frames ? Are the system using JUMBO frames ? If so, are all the network components in the path between those system able to forward jumbo frames ?

What happens, if you try to copy a very small file (1 block) with COPY/FTP ?

Volker.

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 08:20:05 GMT

Hey Volker ,

ok , no mismatches , i did the lan command to an output file on both and searched both.

I tried a copy/ft of a small file as you suggested and it hangs and times out with 'I/O error on network device' , Device Timeout.

I tried a copy without /ftp and it copied across fine.

Hmmm , Is it possible it could be a problem on the actual network switch , might need to get them to check the settings - is there such a thing as 'autonegoutiate' on the port AS WELL AS the server card (which we have verified is ok)

Re: COPY/FTP error

Volker Halle — Wed, 06 Jul 2011 08:28:56 GMT

The switch ports must also be set to auto-negotiate and their current settings should be the SAME as the current settings of the LAN interfaces in your OpenVMS servers.

But if TELNET and DECnet-copy work and FTP copy fails (did you try COPY/FTP x.x localhost::... on the receiving and sending server to make sure the FTP servers are o.k. ?), I guess this might be some problem with a firewall in the path between those servers or a problem with TCP buffer sizes (i.e. the end-to-end buffer size negotiation comes up with a larger buffer size than the intermediate system can handle).

You could run $ TCPTRACE remote-node-IP and/or tcpdump and check the IP packets sent from the local to the remote server.

Volker.

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 09:20:42 GMT

Hi Volker ,

Something interesting - I have another vms server which works fine and is in the same location as the 'receiving server' , when I run the LANCP command at the very end it says

426251871 Packets transmitted (64 bytes)

11034028 Packets transmitted (65-127 bytes)

4253762 Packets transmitted (128-255 bytes)

646907 Packets transmitted (256-511 bytes)

37635 Packets transmitted (512-1023 bytes)

937475 Packets transmitted (1024... bytes)

188557 Multicast packets transmitted

7309565 Broadcast packets transmitted

--- Driver Messages ---

25-OCT-2010 07:24:37.68 Link up: 1000 mbit, full duplex, flow control (receive only)

However when I run the command on both the receiving and sending nodes that we have been looking at the problem on - there is no final 'Driver messages' , all that comes up is

0:00:10.00 Link uptime period to nullify blocking

0:01:00.00 Maximum time to block a link check

--- Fork Delay (after scheduled) ---

--- Transmit Time ---

--- Receive completion time ---

--- One second timer time ---

11 10..19 milliseconds

23 30..39 milliseconds

11 40..49 milliseconds

1 >150 milliseconds

--- Statistics Block ---

10010 Transmits with no carrier sense

May not be relevent but thought Id just mention it having grasped at straws and re checking everything

Re: COPY/FTP error

Volker Halle — Wed, 06 Jul 2011 09:27:27 GMT

OpenVMS Versions and architectures of those 3 servers ?

Does copying a file with COPY/FTP TO the 'good server' work FROM both IBID90 and IBID91 ?

Does COPY/FTP TO IBID90 and IBID91 work FROM the 'good server' ?

Volker.

Re: COPY/FTP error

Hoff — Wed, 06 Jul 2011 09:31:03 GMT

This:

10010 Transmits with no carrier sense

is a problem with the cable or with the host controller or with the switch port.

Usual IP debugging rules apply here:

Trust what the networking folks tell you, but always, always, verify what you're told.
if the networking folks tell you that the switch port or cable is "OK", definitely verify it.
try a different switch port.
try a different cable.
try a different NIC in the box.
always verify the settings on both ends of the connection match.

ftp tends to show errors more than other protocols as it's a glaringly stupid protocol, wildly insecure, and incompatible with inexpensive firewalls. It's also ancient. Given the option, sftp (with or without certificates) is a far better choice.

And in general, using ftp to transfer BACKUP savesets around is asking for trouble. It's one of the most common causes for corrupt metadata. It's better to use zip "-V" (and yes, you need the -V, and yes, you need to quote the -V switch as "-V") to transfer files around, as that preserves the RMS metadata, and most tools can correctly default a zip archive file extension.

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 09:46:35 GMT

Hey Volker ,

unfortunately we dont have decnet configured between the live servers (which was the one I sent the extract from) and the test ones (which we are having the issue on) :(

I have a call out to the network people to check the port settings for me and verify all ok

Re: COPY/FTP error

Volker Halle — Wed, 06 Jul 2011 09:50:23 GMT

COPY/FTP is not DECnet, it's TCPIP. So you can test with this command between those 3 servers...

Do it systematically and report the results.

Volker.

Re: COPY/FTP error

gunners — Wed, 06 Jul 2011 10:11:39 GMT

Something has just become apparent - we have been using the 'hostname' for decnet copies , and the IP address for reflection or keaterm connections - the darn DNS hasnt been updated yet.

I just tried a copy/ftp to the ip address instead , and its working away , sigh....

Thanks everyone for all your help on this issue as always :)