ftp large files fail, small files succeed

[Ron Kaledas]

well, the switch ports that the vms systems are plugged into have been confirmed to be set to 100 full.

ignoring the fact that they should be set to auto (assuming that's safe to ignore for now? the client didn't want to make changes to the prod system) what should I look at next? I saw Andy's note on the traceroute, but I'm not sure what I'm supposed to be looking for there. (though maybe I need to review that note again - darn this not being able to review the thread while replying!)

yes, I know I could cut and paste and such, but it doesn't help when I cut and paste a different message in the thread that I'm replying to. most forums I've been on let you see the whole thread on the same screen as your reply.

anyway, lancp says 100 full, the switch ports say 100 full...thoughts? (I'll review the thread again after I submit this.)

[Hoff]

We've gotten a week along, and I still don't see the VMS versions, which TCP/IP stacks, what the patch levels are, and related details, nor an indicating of whether active or passive has been attempted, nor the contents of any ftp logs, nor information on whether ftp is configured on the target box, nor whether sftp is available and tested.

Try setting the transfer mode.

Post the traceroute.

Post the full command sequence.

Post whether the target box target ftp server is configured and running.

Post logs from the target ftp server.

Post the versions and the patches.

And if that's a full OpenVMS distro kit as it looks, burn a CD and send it over.

Managed switches and VLANs are not error-free and can have issues ranging from configuration errors to wedged ports, and it's also possible for a misconfigured firewall or VLAN to be blocking (for instance) the ephemeral ports.

In this case, also confirm that that ftp is configured and running on the target server, and there's a not a firewall or a VLAN blocking the ephemeral ports. (It's common for managed switches to firewall *everything* unless the boxes have been configured otherwise.

In particular, firewalls and VLANs can play all sorts of games with connectivity, and ftp is about as sensitive as a protocol can be.

Or escalate this internally, or externally.


--

The usual workaround for not being able to view the thread within the ITRC UI is to open another tab with the rest of the thread; Firefox and Safari provide that with version you're likely to have, and IE7 and later can have tabs enabled.

[Jim_McKinney]

You might try using TCPDUMP and watching the FTP control and data connections during a transfer. You may find that the long-idle control connection is terminated by some intervening firewall as Mr Whalen earlier suggested. You may observe retransmissions of packets... perhaps a lossy network. You may see one side or the other stall (as evidenced by connection meta-data) which should direct you to examine processing on that system or prehaps some incongruity with the connection itself... could be lots of things. In any event, I usually find that observing the flow of data is informative and helps get one focusing on where the root of any problem resides if not immediately exposing it.

[Ron Kaledas]

Okay, my mistake for not following up on details, sorry - though I think some of the details requested are actually in the thread above. But, to put it all together:

> We've gotten a week along, and I still don't see the VMS versions, which TCP/IP stacks, what
> the patch levels are, and related details, nor an indicating of whether active or passive has
> been attempted, nor the contents of any ftp logs, nor information on whether ftp is configured
> on the target box, nor whether sftp is available and tested.

passive mode was attempted, with no change in results. I didn't see any ftp logs, the [tcpip$ftp] directory has none. ftp is configured the same on both nodes.
the two nodes are hnaa and hnatst - hnatst was created from a copy of the hnaa system disk, so all patches, etc. are the same on both systems:

hnaa and hnatst patches:

DEC AXPVMS VMS732_PTHREAD V6.0 Patch Install 20-SEP-2007 01:47:36
DEC AXPVMS VMS732_UPDATE V13.0 Patch Install 20-SEP-2007 01:47:05
DEC AXPVMS TCPIP_ECO V5.4-156 Patch Install 20-SEP-2007 01:31:11
DEC AXPVMS VMS732_PCSI V3.0 Patch Install 20-SEP-2007 01:27:46
CPQ AXPVMS CDSA V2.0-109 Full LP Install 20-SEP-2007 01:15:26
DEC AXPVMS DECNET_PHASE_IV V7.3-2 Full LP Install 20-SEP-2007 01:15:26
DEC AXPVMS DWMOTIF V1.3-1 Full LP Install 20-SEP-2007 01:15:26
DEC AXPVMS OPENVMS V7.3-2 Platform Install 20-SEP-2007 01:15:26
DEC AXPVMS TCPIP V5.4-15 Full LP Install 20-SEP-2007 01:15:26
DEC AXPVMS VMS V7.3-2 Oper System Install 20-SEP-2007 01:15:26
HP AXPVMS KERBEROS V2.0-6 Full LP Install 20-SEP-2007 01:15:26

HNAA> tcpip sho ver

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 6
on a AlphaServer ES45 Model 2 running OpenVMS V7.3-2

HNAA>

I am not familiar with sftp, and I don't see it listed in the options for client or server in tcpip$config.


> Try setting the transfer mode.

I believe you're referring to active/passive, if so that didn't help.

> Post the traceroute.

HNATST> traceroute -f 10.252.16.60 1500
traceroute to HNAA (10.252.16.60): 1-30 hops
(initial packetsize = 1500)
1 HNAA (10.252.16.60) 1.66 ms 1.66 ms 1.66 ms
HNATST>

can't get much simpler than that, I imagine!

on a side note, the previously requested ping with packet sizes worked like this:

HNATST> ping -s 65500 10.252.16.60
ping: packet size too large.
%SYSTEM-F-BADPARAM, bad parameter value
HNATST> ping -s 65000 10.252.16.60
PING 10.252.16.60 (10.252.16.60): 65000 data bytes
65008 bytes from 10.252.16.60: icmp_seq=0 ttl=64 time=17 ms
65008 bytes from 10.252.16.60: icmp_seq=1 ttl=64 time=16 ms
65008 bytes from 10.252.16.60: icmp_seq=2 ttl=64 time=15 ms
65008 bytes from 10.252.16.60: icmp_seq=3 ttl=64 time=15 ms


----10.252.16.60 PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms) min/avg/max = 15/16/17 ms
HNATST>

I assume this answers the question as to how much data can be transferred successfully?

> Post the full command sequence.

by this, I assume you meant what I have included in the first post of this thread - were you looking for something more than that?

> Post whether the target box target ftp server is configured and running.

yes, both systems are the same.

> Post logs from the target ftp server.

don't see any logs there either, in sys$sysdevice:[tcpip$ftp]

> Post the versions and the patches.

shown above.

> And if that's a full OpenVMS distro kit as it looks, burn a CD and send it over.

not sure what you're looking for here, I don't have burners on these systems. they are an es45 and an alpha 4100, with 100mb nics.

> Managed switches and VLANs are not error-free and can have issues ranging from configuration
> errors to wedged ports, and it's also possible for a misconfigured firewall or VLAN to be
> blocking (for instance) the ephemeral ports.

I did bring up the firewall issue earlier, but someone didn't think that mattered. I did ask the site to fix the firewall to allow me to connect directly to the test system after I get in via vpn, so maybe that'll help, directly or indirectly. still waiting for that fix to happen.

> In this case, also confirm that that ftp is configured and running on the target server, and
> there's a not a firewall or a VLAN blocking the ephemeral ports. (It's common for managed
> switches to firewall *everything* unless the boxes have been configured otherwise.

> In particular, firewalls and VLANs can play all sorts of games with connectivity, and ftp is
> about as sensitive as a protocol can be.

I will ask them to make sure the ports I'm connected to on the switch will allow everything through.

here's one more sample run, showing a file I was able to transfer, and one that only transferred the filename and size, no data:


HNATST> dir/size=all

Directory SYS$COMMON:[SYSMGR.WEB]

FTP_FAIL.LOG;1 2/18
OPENVMS_ALPHA.ZIP;1
231889/231894

Total of 2 files, 231891/231912 blocks.
HNATST> ftp hnaa
220 HNAA.GCH.GCHOSP.ORG FTP Server (Version 5.4) Ready.
Connected to HNAA.
Name (HNAA:system):
331 Username system requires a Password
Password:
230 User logged in.
FTP> cd sys$common:[sysmgr.web.83]
250-CWD command successful.
250 New default directory is SYS$COMMON:[SYSMGR.WEB.83]
FTP> dir
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSMGR.WEB.83]*.*;* (10.252.16.75,49200)

Directory SYS$COMMON:[SYSMGR.WEB.83]

DEC-AXPVMS-TCPIP-V0506-9ECO5-1.ZIPEXE;1
84846/84852 8-MAR-2010 11:08:27 [SYSTEM] (RWED,RWD,R,R)
OPENVMS_ALPHA_8_3.ZIP;1
409275/409284 13-JUL-2010 10:35:45 [SYSTEM] (RWED,RWED,RWED,)
PATCH_MANIFEST.;1 2/18 13-JUL-2010 09:23:33 [SYSTEM] (RWED,RWD,R,R)
VMS83A_ACRTL-V0600.ZIPEXE;1
6949/6966 1-DEC-2009 18:26:58 [SYSTEM] (RWED,RWD,R,R)
VMS83A_DEBUG-V0200.ZIPEXE;1
8778/8784 29-JAN-2010 18:28:00 [SYSTEM] (RWED,RWD,R,R)
VMS83A_IMGDMP-V0200.ZIPEXE;1
450/450 2-FEB-2010 18:27:52 [SYSTEM] (RWED,RWD,R,R)
VMS83A_PCSI-V0200.ZIPEXE;1
2437/2448 15-APR-2008 19:22:21 [SYSTEM] (RWED,RWD,R,R)
VMS83A_PCSI-V0300.ZIPEXE;1
2456/2466 11-JUL-2010 19:29:50 [SYSTEM] (RWED,RWD,R,R)
VMS83A_SHADOWING-V0600.ZIPEXE;1
1545/1548 3-FEB-2010 18:28:12 [SYSTEM] (RWED,RWD,R,R)
VMS83A_UPDATE-V0900.ZIPEXE;1
99193/99198 18-MAR-2009 19:26:35 [SYSTEM] (RWED,RWD,R,R)
VMS83A_UPDATE-V1000.ZIPEXE;1
100844/100854 9-JUN-2009 19:27:52 [SYSTEM] (RWED,RWD,R,R)
VMS83A_UPDATE-V1200.ZIPEXE;1
102522/102528 17-DEC-2009 18:29:35 [SYSTEM] (RWED,RWD,R,R)

Total of 12 files, 819297/819396 blocks

226 LIST Directory transfer complete.
1612 bytes received in 00:00:00.01 seconds (98.39 Kbytes/s)
FTP> bin
200 TYPE set to IMAGE.
FTP> get patch_manifest.
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSMGR.WEB.83]patch_manifest.; (10.252.16.75,49201) (740 bytes)
226 Transfer complete.
local: SYS$COMMON:[SYSMGR.WEB]PATCH_MANIFEST.;1 remote: patch_manifest.
740 bytes received in 00:00:00.00 seconds (722.66 Kbytes/s)
FTP> get vms83a_imgdmp-v0200.zipexe
200 PORT command successful.
150 Opening data connection for SYS$COMMON:[SYSMGR.WEB.83]vms83a_imgdmp-v0200.zipexe; (10.252.16.75,49202) (230400 bytes)
HNATST::_TNA14: 08:45:51 TCPIP$FTP CPU=00:00:00.66 PF=1522 IO=1151 MEM=342
GET (VMS+) 0 bytes 00:00:02.29 elapsed (0.00 KB/S)
Local: SYS$COMMON:[SYSMGR.WEB]VMS83A_IMGDMP-V0200.ZIPEXE;1
Remote: vms83a_imgdmp-v0200.zipexe
HNATST::_TNA14: 08:48:00 TCPIP$FTP CPU=00:00:00.67 PF=1522 IO=1155 MEM=342
GET (VMS+) 0 bytes 00:02:11.10 elapsed (0.00 KB/S)
Local: SYS$COMMON:[SYSMGR.WEB]VMS83A_IMGDMP-V0200.ZIPEXE;1
Remote: vms83a_imgdmp-v0200.zipexe
%SYSTEM-F-CONNECFAIL, connect to network object timed-out or failed
HNATST> dir/size=all

Directory SYS$COMMON:[SYSMGR.WEB]

FTP_FAIL.LOG;1 2/18
OPENVMS_ALPHA.ZIP;1
231889/231894
PATCH_MANIFEST.;1 2/18
VMS83A_IMGDMP-V0200.ZIPEXE;1
30/450

Total of 4 files, 231923/232380 blocks.
HNATST>
HNATST>

(the ftp_fail.log file just contains the text that I posted in the first message of this thread.)


> Or escalate this internally, or externally.

I didn't think I could call hp support about this, partly because I don't think it's a vms issue.

> --

> The usual workaround for not being able to view the thread within the ITRC UI is to open
> another tab with the rest of the thread; Firefox and Safari provide that with version you're
> likely to have, and IE7 and later can have tabs enabled.

yeah, I know...just an inconvenience, I'll deal with it!

so right now I'm concentrating on the firewall issue. I'll report back what happens when that part gets fixed, if it fixes the ftp issue.

Ron

[Oswald Knoppers_1]

Hi,

In sys$manager on hnaa there should be a TCPIP$FTP_SERVER.LOG file. Is there anything in it?

Can you try getting a trace on hnaa, in a separate session (on hnaa) do:

$ tcptrace/full/port=local=20/prot=tcp/packet=200

Oswald

[Ron Kaledas]

first, an update: they fixed the firewall, I can connect directly to hnatst from the vpn now...but it didn't fix/affect the ftp issue.


so no, that log file in sys$manager ends with a run of tcpip$ftp_child.exe. previous versions of that log file show that and then the logout message.

here's the tcptrace - I ran it on the test system first, it took so long that I was hesitant to run it on prod, not knowing what it's doing to the network:

HNATST> tcptrace/full/port=local=20/prot=tcp/packet=200
HNATST::_TNA15: 09:42:52 TCPIP$TRA CPU=00:00:00.31 PF=853 IO=471 MEM=303
0110FC0A B8160201 00006AA7 1C00C045 0000 E....j..........
00000000 00000000 9BEE6411 | 010000E0 0010 .....d..........
0000 00000000 00000000 00000000 0020 ..............
0110FC0A D9120201 000049AB 1C00C045 0000 E....I..........
00000000 00000000 9BEE6411 | 010000E0 0010 .....d..........
0000 00000000 00000000 00000000 0020 ..............
0110FC0A F80E0201 00002AAF 1C00C045 0000 E....*..........
00000000 00000000 9BEE6411 | 010000E0 0010 .....d..........
0000 00000000 00000000 00000000 0020 ..............

sorry for the formatting. this was still running after 8 minutes, I'm not sure when/if it's going to end...

[Hoff]

> I didn't see any ftp logs, the [tcpip$ftp] directory has none. ftp is configured the same on both nodes.

I had asked if the ftp server was configured and running on the target box, not if it was "the same". If the two hosts are "the same" and "the ftp server is not configured" (which still looks to be in play here), then ftp connections headed the other way may well be broken, too.

That there are no logs in the ftp server directory and in the target user's default directory - the two spots where the logs are expected - implies that the ftp server is not running.

>can't get much simpler than that, I imagine!

Other than localhost, no. That's a simple network.

Given that there are managed switches here, there is a second whole layer of weird in play.

>on a side note, the previously requested ping with packet sizes worked like this:

FWIW, I'd be careful with tossing big pings around. That can crash some boxes, including certain versions of VMS. qv: Ping Of Death.

>> Post the full command sequence.

>by this, I assume you meant what I have included in the first post of this thread - were you looking for something more than that?

Yes, I was wondering about the connect command, and your configuration statement of this-to-that-to-other does make the network appear, um, uniquely configured.

I was also wondering why you were not using the COPY /FTP /BINARY command that's been around in DCL and IP since V6.2, but that's another discussion. Way easier to script that one into a DCL procedure, too. But I digress.

>> Post logs from the target ftp server.

>don't see any logs there either, in sys$sysdevice:[tcpip$ftp]

That's a potential problem. Do look into why you have no logs. A potential trigger here is that the ftp server is not configured and started. And if it's not obvious, the ftp server is different from the ftp client.


>> And if that's a full OpenVMS distro kit as it looks, burn a CD and send it over.

>not sure what you're looking for here, I don't have burners on these systems. they are an es45 and an alpha 4100, with 100mb nics.

OK, translating that apparent confusion, you don't know how to burn optical disk media for use with VMS.

Presuming your software licenses permit it, burning a disk to boot VMS is not particularly difficult, even if you have to haul the disk image over to a Windows or Mac OS X or Linux or Unix box to burn the disk.

At its simplest, you create a 700 MB LD disk partition, BACKUP /IMAGE the distro into it, dismount the LD and copy the backing file over to the box where you have a burner on (if you don't have a burner on VMS, and it's handy to have a burner on VMS even if it's comparatively slow) and burn the disk. Then transport the media. Even with the glacial burn speeds on VMS, I can have a disk generated in well less than a week.

I have directions posted for that elsewhere on the 'net.

http://labs.hoffmanlabs.com/node/28

>I did bring up the firewall issue earlier, but someone didn't think that mattered.

You're thinking of a "firewall" as a device at the network perimeter. That's certainly a common case, but not the only case.

A managed switch can be a very effective firewall.

And it's distinctly possible that various production boxes will have their own firewalls; that's less common with VMS, but it's near-ubiquitous configuration with other platforms (Windows, Mac, Linux, Unix), and it may well be a possibility with VMS guested on a HP-UX box.

This prevalence of firewalls is one of the central reasons why ftp is stinky. Well, that and ftp exposes your access credentials in cleartext.

As for sftp, you type "sftp" at the login prompt. TCP/IP Services versions V5.4 and later have sftp available.

I have various postings on sftp up, including setting up a secure certificate-based, no-password login.

http://labs.hoffmanlabs.com/node/1118

I'd suggest getting an escalation process on-line, too, and ITRC isn't a particularly good one. (I've done hospital IT, and know from experience that the ER docs can get all the way from zero to full cranky milliseconds after the servers or the network have gone walkabout.)

[Oswald Knoppers_1]

>HNATST> tcptrace/full/port=local=20/prot=tcp/packet=200

No no, please do this on HNAA, not on HNATST. Or change the command to /port=remote=20

Oswald

[Ron Kaledas]

quick answer is if ftp server wasn't running, then I wouldn't be able to ftp a small file either.......unless I'm missing something?
longer answer is that ftp under "server options" on both systems is started and enabled.

as you can see below, the log file is configured, I don't know why it's not coming out...

HNAA> tcpip sho serv ftp/full

Service: FTP
State: Enabled
Port: 21 Protocol: TCP Address: 0.0.0.0
Inactivity: 5 User_name: TCPIP$FTP Process: TCPIP$FTP
Limit: 10 Active: 1 Peak: 2

File: TCPIP$SYSTEM:TCPIP$FTP_RUN.COM
Flags: None

Socket Opts: Rcheck Scheck
Receive: 0 Send: 0

Log Opts: Acpt Actv Dactv Conn Error Exit Logi Logo Mdfy Rjct TimO Addr
File: SYS$SYSDEVICE:[TCPIP$FTP]TCPIP$FTP_RUN.LOG

Security
Reject msg: not defined
Accept host: 0.0.0.0
Accept netw: 0.0.0.0
HNAA> dir sys$sysdevice:[tcpip$ftp]

Directory SYS$SYSDEVICE:[TCPIP$FTP]

LOGIN.COM;1 TCPIP$FTP_ANONYMOUS.LOG;1

Total of 2 files.
HNAA>



thanks for the note on big pings...gulp!

disregarding the security issues, I don't see how copy/ftp/binary (which I know about) or sftp would work if ftp doesn't - am I missing something?...sftp does exist but it's not configured.

by the way, I aborted the tcptrace after 20 minutes...not sure how long that was supposed to run or what it was supposed to tell me.

[Ron Kaledas]

>>HNATST> >tcptrace/full/port=local=20/prot=tcp/packet=200

>No no, please do this on HNAA, not on HNATST. >Or change the command to /port=remote=20

>Oswald

okay, but given that I had to abort it after 20 minutes, I'm concerned about its impact on the system, and I don't know how long it's going to run, therefore my hesitation on running it on the prod system or targeted at the prod system...

sorry, I like to understand things fully before I do things on/to prod...