newbie having issues with sftp to VMS 8.3

[Ethan Queen]

I have been having a very annoying probelm when trying to upload files via sftp from multiple Windows machines to a VMS 8.3 system.

Most sftp clients will send the file, but when I open them up on the VMS system, they automagically have pages and pages of blank space in the files.

The blank space isn't added at the end of the file, it is inserted in different areas in the middle of the file. this makes compiling programs impossible until I manually go in and delete all the blank space.

SmartFTP won't even finish sending a file. It dies at ~7k.

I have tried about 8 different sftp clients all with the same results. I have tried encrypted/non-encrypted, binary, text.

I have verified that it has something to do with transferring the file itself, as I can download a clean file, and the re-upload it, and it will grow in size and have blank space as soon as i re-upload it.

What could be the problem?

[Robert Gezelter]

Ethan,

Welcome to the HP ITRC OpenVMS Forum.

Precisely what commands are you using? Are you sure that you are transferring the file as a text file?

Have you tried ZIPing the file on Windows, transferring the ZIP archive, and then UNZIPing the file on OpenVMS?

- Bob Gezelter, http://www.rlgsc.com

[Ethan Queen]

Yes, I know I have tried transferring as both text and binary.

I will try the whole zipping and unzipping thing, but that seems like an extra step/waste of time that I should never have to take.

[RBrown_1]

I don't know anything about sftp, but my impression is that other users of this forum use sftp successfully.

I am intrigued by your description of "blank space" inserted into the middle of binary files. Have you examined the corrupted file with DUMP? Are NULLs inserted? Or blanks? Or s? Is it the same kind of corruption in both text and binary files?

How long does a file have to be to get corrupted? Does an 80-byte (for example) text or binary file get corrupted?

What is the file type and record format of the corrupted file? Is it the same as the original file (in a VMS to windows to VMS transfer sequence)? ($ DIRECTORY/FULL filename).

[Ethan Queen]

00000000 00000000 00000000 00000000
................ 000110

Above is an example of what the blank space looks like when I dump the file.

Also, i just found out by accident that if i upload the same file to a different directory it doesn't add the blank space.

hrmmm.... it looks like it might be something with how versioning is set up. If I rename the same file and then upload it to the original directory it was in, it doesn't add the blank space... even if I upload it multiple times in order to make it add version numbers to the end of the file name.

Versioning is set up so version one ends with ;1, version 2 is ;2, etc.

[Steven Schweda]

> a VMS 8.3 system.

TCPIP SHOW VERSION

or some corresponding thing for whichever
IP product you're using.

> [...] Windows [..]

Which Windows? Which SSH/SFTP software?

> SmartFTP [...]

There's more than one version of that, too.

And probably more than one way to do anything
with any of them.

> [...] pages and pages of blank space in the
> files.

Not a particularly useful description.

DIRE /FULL destination_file

DUMP might reveal something interesting, but
perhaps not worth the bother.

> I have tried about 8 different sftp clients
> [...]

Again, not a particularly useful description,
but it would suggest that the problem may lie
with the server.

> What could be the problem?

Almost anything?

Around here, I lack Windows, but I had no
trouble moving a text file of some size from
an HP-UX system to my main VMS system:

dyi # sftp sms@alp
Connecting to alp...

@ SYS$MANAGER:ANNOUNCE.TXT
sftp> put zipfile.c
Uploading zipfile.c to /ALP$DKA0/SMS/zipfile.c
zipfile.c 100% 220KB 73.4KB/s 96.0KB/s 00:03
sftp> quit
dyi #

alp $ diff zipfile.c ALP$DKA0:[UTILITY.SOURCE.ZIP.zip31c02]
Number of difference sections found: 0
Number of difference records found: 0

DIFFERENCES /IGNORE=()/MERGED=1-
ALP$DKA0:[SMS]ZIPFILE.C;1-
ALP$DKA0:[UTILITY.SOURCE.ZIP.ZIP31C02]ZIPFILE.C;2

(Which is where it came from originally.)


dyi # uname -a
HP-UX dyi B.11.31 U ia64 4235313755 unlimited-user license

dyi # ssh -V
OpenSSH_5.2p1+sftpfilecontrol-v1.3-hpn13v5, OpenSSL 0.9.8k 25 Mar 2009
HP-UX Secure Shell-A.05.20.015, HP-UX Secure Shell version

ALP $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 5
on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3

alp $ ssh "-V"
alp$dka0:[sys0.syscommon.][sysexe]tcpip$ssh_ssh2.exe: SSH Secure Shell OpenVMS (
V5.5) 3.2.0 on COMPAQ Professional Workstation - VMS V8.3

[Steven Schweda · Accepted solution]

> Versioning is set up so version one ends
> with ;1, version 2 is ;2, etc.

Yeah, that's VMS.

> [...] If I rename the same file [...]

Ah. Not unusual behavior for a UNIX-like
program on a VMS system. I ran into it with
GnuPG. As the notes there say, ...

[...]
GPG normally resists overwriting an existing output file, prompting
an interactive user with a question like: "File `name.type' exists.
Overwrite? (y/N)". "Overwrite" may appear to be safe, because a new
file version will be created instead of actually overwriting the
existing file. However, if the existing file has non-UNIX-like
attributes ("Record format: Variable length", for example), then the new
file will inherit the attributes of the old file, and the result may be
a corrupt file, because the UNIX-like GPG code writes its output in ways
which do not conform to the original file's record structure.
Overwriting a Stream_LF file should be harmless.
[...]

I haven't observed this when using SCP/SFTP
(et al.), but it sure sounds like the same
syndrome. Everything's complicated.


http://antinode.info/ftp/gnupg/gnupg-1_4_10a_vms/vms_notes.txt

[Ethan Queen]

HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.6 - ECO 2
on an HP rx2600 (1.40GHz/1.5MB) running OpenVMS V8.3

I have tried from both XP Pro 32-bit and Windows 7 Ultimate x64.

SSH/SFTP software I have tried (all latest versions)

WinSCP 4.2.7
Bitvise Tunnelier 4.33
BitKinex 3.1.1
AnyClient 2.0
FileZilla 3.3.2
SmartFTP 4.0.1108.0

I think there was 1 or 2 more, but I unustalled them and deleted the installers because the interface was horrid.

NEWCGI.C;688 File ID: (42787,38,0)
Size: 65/72 Owner: [CSM,KAYCEE]
Created: 10-MAY-2010 16:09:56.75
Revised: 8-MAY-2010 19:18:05.00 (1)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 72, Extend: 0, Global buffer count: 0
No version limit
Record format: Variable length, maximum 0 bytes, longest 108 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RWED, World:R
Access Cntrl List: None
Client attributes: None

The second part where yo mention about it inheriting the file attributes kind of makes sense, but doesn't since the file will corrupt if I download it and then re-upload it without changing anything.

[Hoff]

I suspect the problem is encoding or file format.

Ok, you're on Windows, which means you have RTF and a whole pile of Microsoft formats and encodings at your disposal, and who knows what format this source code is in.

Sequential formats are sometimes transferable, but surprisingly often not. I'm wondering if the file formats involved here on Windows just aren't (entirely) readable by OpenVMS using its standard tools.

And there are about a gazillion encodings around, and VMS is only marginally able to deal with ISO Latin 1 (ISO-8859-1) at best; that's fairly similar to MCS. Get one of the UTF encodings or RTF formats in play, and all bets are off.

What Microsoft calls MS-DOS line endings or a stream file are probably the closest encodings to what OpenVMS expects.

I'd suggest the xxd tool or other such file dump tools on the source box and have a look at the first block or two of the source file on the originating platform, but this is Microsoft Windows and the command-level tools over there are increasingly unfamiliar.

And FWIW, one of the few Microsoft Windows file transfer tools that deals reasonably well with the oddities around OpenVMS is Filezilla. But my guess here is that you're getting in trouble at a lower level than the transfer itself.

You mention differences when pulling down and then pushing up these source files; are you doing that so that you can edit the files? Or is there some other reason? And what tools are editing and modifying the files during their stay on the Windows box?

[Steven Schweda]

> [...] Version V5.6 - ECO 2 [...]

Not the latest, but I wouldn't expect a newer
one to fix this problem.

> Record format: Variable length, [...]

That would do it. Like GnuPG, I'd guess that
the SFTP server thinks that (or acts as if)
it's writing a Stream_LF file (which, I'd
guess, is what you get if there's no existing
file).

> [...] but doesn't since the file will
> corrupt if I download it and then re-upload
> it without changing anything.

I don't know which way is up (or down). As
(I understand what) you said, if you create a
fresh file (and not a new version of an
existing non-Stream_LF file), then it works,
and if you create a new version of an
existing non-Stream_LF file, then it fails.
That sounds to me exactly like what I said.