newbie having issues with sftp to VMS 8.3

[Ethan Queen]

I have been having a very annoying probelm when trying to upload files via sftp from multiple Windows machines to a VMS 8.3 system.

Most sftp clients will send the file, but when I open them up on the VMS system, they automagically have pages and pages of blank space in the files.

The blank space isn't added at the end of the file, it is inserted in different areas in the middle of the file. this makes compiling programs impossible until I manually go in and delete all the blank space.

SmartFTP won't even finish sending a file. It dies at ~7k.

I have tried about 8 different sftp clients all with the same results. I have tried encrypted/non-encrypted, binary, text.

I have verified that it has something to do with transferring the file itself, as I can download a clean file, and the re-upload it, and it will grow in size and have blank space as soon as i re-upload it.

What could be the problem?

[Robert Gezelter]

Ethan,

Welcome to the HP ITRC OpenVMS Forum.

Precisely what commands are you using? Are you sure that you are transferring the file as a text file?

Have you tried ZIPing the file on Windows, transferring the ZIP archive, and then UNZIPing the file on OpenVMS?

- Bob Gezelter, http://www.rlgsc.com

[Ethan Queen]

Yes, I know I have tried transferring as both text and binary.

I will try the whole zipping and unzipping thing, but that seems like an extra step/waste of time that I should never have to take.

[RBrown_1]

I don't know anything about sftp, but my impression is that other users of this forum use sftp successfully.

I am intrigued by your description of "blank space" inserted into the middle of binary files. Have you examined the corrupted file with DUMP? Are NULLs inserted? Or blanks? Or s? Is it the same kind of corruption in both text and binary files?

How long does a file have to be to get corrupted? Does an 80-byte (for example) text or binary file get corrupted?

What is the file type and record format of the corrupted file? Is it the same as the original file (in a VMS to windows to VMS transfer sequence)? ($ DIRECTORY/FULL filename).

[Ethan Queen]

00000000 00000000 00000000 00000000
................ 000110

Above is an example of what the blank space looks like when I dump the file.

Also, i just found out by accident that if i upload the same file to a different directory it doesn't add the blank space.

hrmmm.... it looks like it might be something with how versioning is set up. If I rename the same file and then upload it to the original directory it was in, it doesn't add the blank space... even if I upload it multiple times in order to make it add version numbers to the end of the file name.

Versioning is set up so version one ends with ;1, version 2 is ;2, etc.

[Steven Schweda]

> a VMS 8.3 system.

TCPIP SHOW VERSION

or some corresponding thing for whichever
IP product you're using.

> [...] Windows [..]

Which Windows? Which SSH/SFTP software?

> SmartFTP [...]

There's more than one version of that, too.

And probably more than one way to do anything
with any of them.

> [...] pages and pages of blank space in the
> files.

Not a particularly useful description.

DIRE /FULL destination_file

DUMP might reveal something interesting, but
perhaps not worth the bother.

> I have tried about 8 different sftp clients
> [...]

Again, not a particularly useful description,
but it would suggest that the problem may lie
with the server.

> What could be the problem?

Almost anything?

Around here, I lack Windows, but I had no
trouble moving a text file of some size from
an HP-UX system to my main VMS system:

dyi # sftp sms@alp
Connecting to alp...

@ SYS$MANAGER:ANNOUNCE.TXT
sftp> put zipfile.c
Uploading zipfile.c to /ALP$DKA0/SMS/zipfile.c
zipfile.c 100% 220KB 73.4KB/s 96.0KB/s 00:03
sftp> quit
dyi #

alp $ diff zipfile.c ALP$DKA0:[UTILITY.SOURCE.ZIP.zip31c02]
Number of difference sections found: 0
Number of difference records found: 0

DIFFERENCES /IGNORE=()/MERGED=1-
ALP$DKA0:[SMS]ZIPFILE.C;1-
ALP$DKA0:[UTILITY.SOURCE.ZIP.ZIP31C02]ZIPFILE.C;2

(Which is where it came from originally.)


dyi # uname -a
HP-UX dyi B.11.31 U ia64 4235313755 unlimited-user license

dyi # ssh -V
OpenSSH_5.2p1+sftpfilecontrol-v1.3-hpn13v5, OpenSSL 0.9.8k 25 Mar 2009
HP-UX Secure Shell-A.05.20.015, HP-UX Secure Shell version

ALP $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.6 - ECO 5
on a COMPAQ Professional Workstation XP1000 running OpenVMS V8.3

alp $ ssh "-V"
alp$dka0:[sys0.syscommon.][sysexe]tcpip$ssh_ssh2.exe: SSH Secure Shell OpenVMS (
V5.5) 3.2.0 on COMPAQ Professional Workstation - VMS V8.3

[Steven Schweda · Accepted solution]

> Versioning is set up so version one ends
> with ;1, version 2 is ;2, etc.

Yeah, that's VMS.

> [...] If I rename the same file [...]

Ah. Not unusual behavior for a UNIX-like
program on a VMS system. I ran into it with
GnuPG. As the notes there say, ...

[...]
GPG normally resists overwriting an existing output file, prompting
an interactive user with a question like: "File `name.type' exists.
Overwrite? (y/N)". "Overwrite" may appear to be safe, because a new
file version will be created instead of actually overwriting the
existing file. However, if the existing file has non-UNIX-like
attributes ("Record format: Variable length", for example), then the new
file will inherit the attributes of the old file, and the result may be
a corrupt file, because the UNIX-like GPG code writes its output in ways
which do not conform to the original file's record structure.
Overwriting a Stream_LF file should be harmless.
[...]

I haven't observed this when using SCP/SFTP
(et al.), but it sure sounds like the same
syndrome. Everything's complicated.


http://antinode.info/ftp/gnupg/gnupg-1_4_10a_vms/vms_notes.txt

[Ethan Queen]

HP TCP/IP Services for OpenVMS Industry Standard 64 Version V5.6 - ECO 2
on an HP rx2600 (1.40GHz/1.5MB) running OpenVMS V8.3

I have tried from both XP Pro 32-bit and Windows 7 Ultimate x64.

SSH/SFTP software I have tried (all latest versions)

WinSCP 4.2.7
Bitvise Tunnelier 4.33
BitKinex 3.1.1
AnyClient 2.0
FileZilla 3.3.2
SmartFTP 4.0.1108.0

I think there was 1 or 2 more, but I unustalled them and deleted the installers because the interface was horrid.

NEWCGI.C;688 File ID: (42787,38,0)
Size: 65/72 Owner: [CSM,KAYCEE]
Created: 10-MAY-2010 16:09:56.75
Revised: 8-MAY-2010 19:18:05.00 (1)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 72, Extend: 0, Global buffer count: 0
No version limit
Record format: Variable length, maximum 0 bytes, longest 108 bytes
Record attributes: Carriage return carriage control
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:RWED, World:R
Access Cntrl List: None
Client attributes: None

The second part where yo mention about it inheriting the file attributes kind of makes sense, but doesn't since the file will corrupt if I download it and then re-upload it without changing anything.

[Hoff]

I suspect the problem is encoding or file format.

Ok, you're on Windows, which means you have RTF and a whole pile of Microsoft formats and encodings at your disposal, and who knows what format this source code is in.

Sequential formats are sometimes transferable, but surprisingly often not. I'm wondering if the file formats involved here on Windows just aren't (entirely) readable by OpenVMS using its standard tools.

And there are about a gazillion encodings around, and VMS is only marginally able to deal with ISO Latin 1 (ISO-8859-1) at best; that's fairly similar to MCS. Get one of the UTF encodings or RTF formats in play, and all bets are off.

What Microsoft calls MS-DOS line endings or a stream file are probably the closest encodings to what OpenVMS expects.

I'd suggest the xxd tool or other such file dump tools on the source box and have a look at the first block or two of the source file on the originating platform, but this is Microsoft Windows and the command-level tools over there are increasingly unfamiliar.

And FWIW, one of the few Microsoft Windows file transfer tools that deals reasonably well with the oddities around OpenVMS is Filezilla. But my guess here is that you're getting in trouble at a lower level than the transfer itself.

You mention differences when pulling down and then pushing up these source files; are you doing that so that you can edit the files? Or is there some other reason? And what tools are editing and modifying the files during their stay on the Windows box?

[Steven Schweda]

> [...] Version V5.6 - ECO 2 [...]

Not the latest, but I wouldn't expect a newer
one to fix this problem.

> Record format: Variable length, [...]

That would do it. Like GnuPG, I'd guess that
the SFTP server thinks that (or acts as if)
it's writing a Stream_LF file (which, I'd
guess, is what you get if there's no existing
file).

> [...] but doesn't since the file will
> corrupt if I download it and then re-upload
> it without changing anything.

I don't know which way is up (or down). As
(I understand what) you said, if you create a
fresh file (and not a new version of an
existing non-Stream_LF file), then it works,
and if you create a new version of an
existing non-Stream_LF file, then it fails.
That sounds to me exactly like what I said.

[Ethan Queen]

Yeah, I thought about it more.. and what you said makes sense.

Is there any way to change the record type of an existing file? If so, can I change all the files in a directory at the same time?

[Steven Schweda]

If I use a text editor to create a "Record
format: Variable length" file on my VMS
system, and then use SFTP to "overwrite" it,
I get junk (as expected/feared):

alp $ rename ZIPFILE.C;* ZIPFILE.C_one;*
alp $ edit /tpu ZIPFILE.C
[...]

dyi # sftp sms@alp
Connecting to alp...

@ SYS$MANAGER:ANNOUNCE.TXT
sftp> put zipfile.c
Uploading zipfile.c to /ALP$DKA0/SMS/zipfile.c
zipfile.c 100% 220KB 73.4KB/s 96.0KB/s 00:03
sftp> quit
dyi #


alp $ diff zipfile.c ALP$DKA0:[UTILITY.SOURCE.ZIP.zip31c02]
%DIFF-F-READERR, error reading ALP$DKA0:[SMS]ZIPFILE.C;2
-RMS-W-RTB, 30068 byte record too large for user's buffer

Looks (to me) exactly like the GnuPG
syndrome.

> [...] download [...] re-upload [...]

If you fetch a "Record format: Variable
length" file from the VMS system, the SFTP
server may be able to _read_ the thing
properly, so you could get good text at a
non-VMS system, but that won't help you if
you ask the SFTP server to "overwrite" the
thing on the VMS server. This is one of the
problems with using UNIX-like code on a VMS
system.

In my case, this is an ODS5 disk, and the
SFTP server doesn't even preserve the
original (lower) case of the file name. When
dealing with TCPIP, low expectations can be
more realistic than high ones.

[Ethan Queen]

Hoff - I am using VS-2010 to do the editing, but as i said before, it doesn't matter if I even edit the files or not.

[Steven Schweda]

> Is there any way to change the record type
> of an existing file?

You can convert it. I use this:

ALP $ type utility:cnvstmlf.com
$! 12 December 1999. SMS.
$!
$! CONVERT a file to StreamLF record format.
$!
$ convert 'p1' 'p2' /fdl = sys$input:
RECORD
FORMAT stream_lf
$!


I usually do something like:

@ utility:cnvstmlf.com zipfile.c zipfile.c

Note that that'll create a new version of the
old file (with the desired record format),
without destroying the original.

> If so, can I change all the files in a
> directory at the same time?

I could write a short procedure, but I know
of no single command.

[Robert Gezelter]

Ethan,

I don't have Visual Studio loaded here at the moment. In Word, the file type you are looking for is "Plain Text".

One way is to look at the file from an C:\> prompt on Windows. Use the more utility, to wit:

more
If the file has a lot of nonsense, then it is one of the non-text Microsoft formats (.DOC, .DOCX, and a large number of others). By contrast, a plain text file will display correctly using "more").

I would create a test file using Word or Wordpad, saving the file as "Plain Text" and try uploading that to the OpenVMS system as a test.

I also seem to recall that there have been some tools done to ease development in this context, but they may not be justified by your situation.

- Bob Gezelter, http://www.rlgsc.com

[Hoff]

Getting the Microsoft IDE involved is probably not going to have the most desired outcome barring aftermarket format conversions; the Microsoft native file formats and encodings (with the exception of a couple of the most primitive among these) are generally not compatible with OpenVMS.

NetBeans and Eclipse are IDE options for use with OpenVMS.

The former is provided by HP, the latter by eCube Systems.

Other possibilities (and I've not checked to see if either of these provides the requisite conversions) is the Windows client for DECset CMS, and there's always Mercurial.

[Ethan Queen]

The MS Visual Studio IDE save pretty much in plain text format as far as I can tell.

I very much doubt it would get much use at all if it didn't as most files need to be cross-platform and not have any issues with opening on different systems or in different editors/IDEs.

Using more on a file saved with VS2010 dispalys properly.

With the MS IDE, I have pretty much all I need which is syntax checking for whatever files I am working on.... html, js, c, css, etc. And it has a nice layout and very good funtionality.

As for NetBeans or Eclipse... i have tried them in the past and do not like them as much as the MS IDE.. and NetBeans is written in Java... and though it may be better then it used to be, I am not too keen on using a Java based IDE.

Thanks for all the help and info. I am going to see if we can set the server up to auto change the file to stream-lf when saved on the server.

Other then that, everything is working great.

[Steven Schweda]

> [...] I use this:
> [...]

To find them, I use this:

ALP $ type utility:non-stmlf.com
$! 9 November 2006. SMS.
$!
$! Find non-directory files matching P1 which are not
$! Record format: Stream_LF.
$!
$!
$ if (p1 .eqs. "")
$ then
$ p1 = "[...]*.*;*"
$ endif
$!
$ file_old = ""
$ loop_top:
$!
$ file = f$search( p1)
$ if ((file .eqs. "") .or. (file .eqs. file_old)) then goto loop_end
$!
$ file_old = file
$ if (.not. f$file_attributes( file, "DIRECTORY"))
$ then
$!
$ rfm = f$file_attributes( file, "RFM")
$ if (rfm .nes. "STMLF")
$ then
$ write sys$output "''rfm' ''file'"
$ endif
$!
$ endif
$!
$ goto loop_top
$!
$ loop_end:
$!
ALP $

which could be adapted to do some automatic
conversion, if you're careful.


When I work on an open-source project, I try
to remember to keep all my source files in
Stream_LF format, to avoid problems with
portability to non-VMS systems, lame Web
servers, lame FTP servers, and so on. In
general, programs like UnZip and VMSTAR will
(or can be persuaded to) create Stream_LF
files. The problem comes when using a text
editor to create a new file. I've tried to
train myself to copy an old (Stream_LF) file
when creating a new one, but when I forget to
do that, I (hope to remember to) run
non-stmlf.com on the whole mess to see what I
missed. It'd be nice if there were a simple
way to get an editor to default to the
preferred record format, but I'm unaware of
any.

[Hoff]

The statement "The MS Visual Studio IDE save pretty much in plain text format as far as I can tell." (sorry!) and the reported symptoms and the use of Microsoft Windows software and tools taken together can imply that the formats and encodings are anything but plain text.

Look at the file formats and at the character encodings.

This consideration is something that isn't necessary when doing work on homogeneous platforms, but becomes an issue when moving files around.

If you toss a UTF-8 linefeed-terminated sequential file at OpenVMS (stream LF records), it'll work until it hits some Unicode and it'll spew. Microsoft Windows knows from newer formats and encodings. OpenVMS doesn't.

Create a simple Hello, World with a half dozen to a dozen lines of C on VMS using VMS tools; something small and manageable.

Transfer it.

Rustle up a port of xxd or Cygwin with xxd or a Windows native data dump tool and have a look at the data in the file as it arrives on the Windows box. Compare that with the source dump using the VMS DUMP command.

You'll also want to use a file differences tool to compare the file transferred onto Windows with a copy of the file saved out of the Windows application.

Yes, both NetBeans and Eclipse are Java. They do both work with VMS. I haven't heard of a plug-in for the Microsoft IDE for doing OpenVMS work.

[Steven Schweda]

> The statement [...]

_Whose_ statement?

> [...] the reported symptoms [...]

Reported by whom?

I've shown how to produce what appears to be
similar file corruption without involving any
Windows system or any Microsoft software,
only an HP-UX system and a plain ".c" file.

While it may be easy to find exotic file
formats on a Windows system, I really don't
see any hint here (from the fellow with the
problem) that he's dealing with anything
other than plain text. You certainly don't
need anything exotic to get the TCPIP SFTP
server to corrupt a file this way.

You can bark up any tree you choose, but I
think that mine has a tastier critter in it.

[Steven Schweda]

> [...] an HP-UX system [...]

Note that using only a VMS system, you may
not see the same problem. The debug info
shows stuff like:

Ssh2SftpServer/SSHFILEXFERS.C:3484: extended attribute name: rms-rfmt-data@openvms.hp.com alq=0, lrl=0, mrs=0, rfm=var, rat=cr

and the result is another rfm=var file.

Coming from a non-VMS system (like, say,
HP-UX, or even Windows), the TCPIP SFTP
server will (apparently) do STREAM_LF-like
I/O, but that doesn't work well when the file
inherits the non-STREAM_LF attributes of an
existing "Variable length" file (soon to
become ";-1").

Everything's complicated (even without
Microsoft).

[Ethan Queen]

O.k., so I did some research and testing and came up with this to go through and convert any file (newest versions only) to STREAM_LF.

The search script that was posted returned any version of the file that was not stream_lf and when I added CONVERT to it, and ran it, it made a new file.

But, because it was searching for any version of the file that was not stream_lf, it would make a new file every time I ran it.

Here is the script:

$! 9 November 2006. SMS.
$! Changes found files to Stream_LF format - 16 May 2010. EQ.
$!
$! Find non-directory files matching P1 which are not
$! Record format: Stream_LF.
$!
$!
$ if (p1 .eqs. "")
$ then
$ p1 = "[...]*.*;*"
$ endif
$!
$ file = f$search(p1)
$ file_old = file
$ file_name = (f$parse(file,,,"NAME")-"^;")
$ file_name_old = file_name
$!
$ loop_top:
$!
$ file = f$search(p1)
$! write sys$output "''FID' ''file'"
$ file_name = (f$parse(file,,,"NAME")-"^;")
$ if (file_name_old .eqs. "") then goto loop_end
$ if (file_name .eqs. file_name_old) then goto loop_top
$!
$ if (.not. f$file_attributes( file_old, "DIRECTORY"))
$ then
$!
$ rfm = f$file_attributes( file_old, "RFM")
$ if (rfm .nes. "STMLF")
$ then
$ write sys$output "''FID' ''file_old'"
$ CONVERT/FDL=STREAM_LF.FDL 'file_name_old' 'file_name_old'
$ endif
$!
$ endif
$ file_old = file
$ file_name_old = file_name
$!
$ goto loop_top
$!
$ loop_end:
$!

And you will also need to make a STREAM_LF.FDL file with this in it:

FILE
ALLOCATION 4
BEST_TRY_CONTIGUOUS yes
EXTENSION 0
ORGANIZATION sequential

RECORD
BLOCK_SPAN yes
FORMAT stream_LF
SIZE 0

[Ethan Queen]

Slight bug in the script I posted. This should be fixed to work properly now:

$! 9 November 2006. SMS.
$! Changes found files to Stream_LF format - 16 May 2010. EQ.
$!
$! Find non-directory files matching P1 which are not
$! Record format: Stream_LF.
$!
$!
$ if (p1 .eqs. "")
$ then
$ p1 = "[...]*.*;*"
$ endif
$!
$ file = f$search(p1)
$ file_old = file
$ file_name = f$parse(file,,,"NAME")
$ file_name_old = file_name
$!
$ loop_top:
$!
$ file = f$search(p1)
$! write sys$output "''FID' ''file'"
$ file_name = f$parse(file,,,"NAME")
$ if (file_name_old .eqs. "") then goto loop_end
$ if (file_name .eqs. file_name_old) then goto loop_top
$!
$ if (.not. f$file_attributes( file_old, "DIRECTORY"))
$ then
$!
$ rfm = f$file_attributes( file_old, "RFM")
$ if (rfm .nes. "STMLF")
$ then
$ write sys$output "''FID' ''file_old'"
$ CONVERT/FDL=STREAM_LF.FDL 'file_old' 'file_old'
$ endif
$!
$ endif
$ file_old = file
$ file_name_old = file_name
$!
$ goto loop_top
$!
$ loop_end:
$!